Belief Updating in Moral Dilemmas

Zachary Horne & Derek Powell & Joseph Spino

Published online: 22 September 2013# Springer Science+Business Media Dordrecht 2013

Abstract Moral psychologists have shown that people’s past moral experiences canaffect their subsequent moral decisions. One prominent finding in this line of research isthat when people make a judgment about the Trolley dilemma after considering theFootbridge dilemma, they are significantly less likely to decide it is acceptable to redirecta train to save five people. Additionally, this ordering effect is asymmetrical, as making ajudgment about the Trolley dilemma has little to no effect on people’s judgments aboutthe Footbridge dilemma. We argue that this asymmetry is the result of a difference inhow each dilemma affects people’s beliefs about the importance of saving lives. In twoexperiments, we show that considering the Footbridge dilemma disconfirms thesebeliefs, while considering the Trolley dilemma does not significantly affect them.Consistent with predictions of sequential learning models, our findings offer a clearand parsimonious account of the asymmetry in the ordering effect.

A growing body of research in moral psychology has shown that people’s past moralexperiences can affect their subsequent moral decisions (Liao et al. 2011; Lombrozo2009; Sinnott-Armstrong 2008; Wiegmann et al. 2008; Wiegmann and Okan 2012).One prominent finding in this line of research is that when people make a judgmentabout the Trolley dilemma after considering the Footbridge dilemma, they are signifi-cantly less likely to decide it is acceptable to redirect a train in order to save five people.1

Additionally, this ordering effect is asymmetrical. Whereas making a judgment about

Rev.Phil.Psych. (2013) 4:705–714DOI 10.1007/s13164-013-0159-y

1In the Trolley dilemma, participants are told they are at the wheel of a runaway train that will cause thedeaths of five workmen if it proceeds on its present course. The only way to avoid the deaths of theseworkmen is to hit a switch, redirecting the train to a side track where it will kill a single workman instead.This dilemma is often contrasted with another famous moral dilemma, the Footbridge dilemma, which asksparticipants to consider a situation in which they are on a footbridge, in between a runaway trolley and fiveworkmen who will be killed if nothing is done. Participants are then told that the only way to save the livesof the five workmen is to push a stranger off the bridge and onto the tracks below where his large body willstop the trolley.

Z. Horne (*) : J. SpinoDepartment of Philosophy, University of Illinois - Urbana-Champaign, Urbana, USAe-mail: horne2@illinois.edu

D. PowellDepartment of Psychology, University of California - Los Angeles, Los Angeles, USA

the Footbridge dilemma affects people’s judgments about the Trolley dilemma, makinga judgment about the Trolley dilemma has little to no effect on their judgments about theFootbridge dilemma (Schwitzgebel and Cushman 2012). We call this the Footbridge-Trolley ordering effect.

Some have argued that the Footbridge-Trolley ordering effect provides critical insightinto the mental processes that underlie people’s moral judgments (Schwitzgebel andCushman 2012), while others have interpreted this finding as being substantial evidenceagainst the use of intuitions in ethical theorizing (Liao et al. 2011; Sinnott-Armstrong2008; Tobia et al. 2012). Despite extensive empirical research and philosophicaldiscussion of this effect, there is no consensus explanation as to why it occurs.

Researchers have proposed a number of explanations for ordering effects on moraljudgments. Some researchers suggest that these ordering effects are produced whenparticipants compare similar moral vignettes (Liao et al. 2011; Sinnott-Armstrong 2008;Wiegmann et al. 2008). It seems likely that a process of comparison plays a role inproducing ordering effects. However, this suggestion does not offer an explanation ofwhy the Footbridge-Trolley ordering effect is asymmetrical. Given that the dilemmas areso similar, it is unclear why comparing the Footbridge dilemma to the Trolley dilemmaaffects people’s judgments about the Trolley dilemma but not vice versa.

Schwitzgebel and Cushman (2012) have proposed that this asymmetry is explainedby a difference in the mental processes that are recruited when people make utilitarian ordeontological moral judgments. These researchers assume a dual-process account ofmoral reasoning wherein different mental processes produce different types of moraljudgments; emotional processing leads to deontological moral judgment whereas rea-soning processes lead to utilitarian moral judgments. Schwitzgebel and Cushman arguethat the emotional processes typically associated with deontological moral judgment arenot sensitive to new evidence, such as a previously considered moral dilemma. Incontrast, the reasoning processes typically associated with utilitarian moral judgmentare responsive to new evidence. This would explain why people’s judgments about theTrolley dilemma (utilitarian) are influenced by their previous judgment about theFootbridge dilemma (deontological), but that the converse does not hold.

As noted, Schwitzgebel and Cushman's account presumes that the Trolley andFootbridge dilemmas recruit fundamentally different mental processes. Cast in this light,the asymmetry in the ordering effect may be construed as additional evidence thatdifferent mental processes underlie deontological and utilitarian moral judgment.However, asymmetrical ordering effects have been widely reported in research onjudgments of non-moral problems, and have been explained without appeal to differ-ences in mental processes (Hogarth and Einhorn 1992). Moreover, some researchershave found evidence that moral judgments recruit domain general cognitive processessuch as decision-making heuristics (Rai and Holyoak 2010), and mechanisms associatedwith causal reasoning (Waldmann et al. 2012). In contrast to Schwitzgebel andCushman’s proposal, these researchers argue that the field should seek domain-general explanations of moral behavior before positing more specialized mental pro-cesses that appeal specifically to the details of the dilemmas under consideration.

In the present paper we sought to examine whether domain general models oflearning and judgment can provide insight into the Footbridge-Trolley orderingeffect. Sequential learning models have been developed across a variety of domains,describing how people and (non-human animals) form stimulus–response

706 Z. Horne et al.

associations (Rescorla and Wagner 1972), how people learn causal relationships(Shanks 1985; Van Hamme and Wasserman 1994; Danks et al. 2003), and how theyupdate their beliefs (Hogarth and Einhorn 1992). Sequential learning models describehow the state of some mental representation is affected by new experiences. In manymodels, the possible states of these mental representations are bounded their minimumand maximum states are often expressed as probabilities of 0 and 1 (Danks et al. 2003).

In order to constrain the possible states of mental representations within sequentiallearning models, the degree to which some experience affects a given mental repre-sentation is contingent on the current state of that representation. Specifically, theweight assigned to new experiences on a given mental representation is inverselyproportional to the difference from the representation’s current state to whicheverbound it is approaching. So when a belief or association is strong, experiencessupporting the representation are less influential than experiences that undermine it.And just the opposite is true when the representation is weak. This dynamic can leadto asymmetrical ordering effects.

We propose that sequential learning models can explain the Footbridge-Trolleyordering effect. In the next section, we consider Hogarth and Einhorn’s (1992) Belief-Adjustment model, a sequential learning model that was explicitly developed toexplain ordering effects. However, as discussed above, the general principles wesummarized are at work in a number of models that describe a wide range of human(and nonhuman animal) behavior (Danks et al. 2003; Hogarth and Einhorn 1992;Rescorla and Wagner 1972; Shanks 1985; Van Hamme and Wasserman 1994).

1 A Belief-Adjustment Explanation

Hogarth and Einhorn’s Belief-Adjustment model (1992) specifies how the state ofsome belief is affected by new evidence. The following equation describes the model:

Sk ¼ Sk−1 þ wk s xkð Þ−R½ �:

Sk is the state of some belief after k pieces of evidence and Sk−1 is the state of thatbelief just before the kth piece of evidence. The variable s(xk) is the individual’ssubjective evaluation of the kth piece of evidence. The variable R is a reference pointagainst which the kth piece of evidence is evaluated and wk is the weight assigned tothe difference between the individual’s subjective evaluation and their referencepoint. Each of these terms takes on a value between 0 and 1.

To help frame the Footbridge-Trolley ordering effect in the context of prior researchon ordering effects, we must consider what belief, if any, is adjusted when peopleconsider these moral dilemmas. For naive participants, Sk−1 cannot be about either theTrolley or the Footbridge dilemmas in particular, since participants have never consid-ered either dilemma prior to participating in the experiment. Rather, Sk−1 is a generalmoral belief that is recruited when one considers either of these moral situations. Forexample, a candidate belief might be “In life or death situations, one should always takewhatever means necessary to save the most lives.” In this case, participants’ judgmentsabout a dilemma constitute their evaluation of a new piece of evidence s(xk) that isrelevant to their more general belief Sk−1. Hogarth and Einhorn’s model states how these

Belief Updating in Moral Dilemmas 707

evaluations affect Sk−1, resulting in an updated belief Sk. One standing issue is why theFootbridge and Trolley dilemmas affect some belief Sk−1 to different degrees.

As in other sequential learning models, in the Belief-Adjustment model the effectof evidence is dependent on the current state of the belief, S. In the Belief-Adjustmentmodel, when evidence is disconfirmatory the weight is

wk ¼ αSk−1;

whereas if evidence is confirmatory, the weight is

wk ¼ β 1−Sk−1ð Þ:Thus, the weight of disconfirmatory evidence is proportional to the strength of the

belief (Sk−1), whereas the weight of confirmatory evidence is proportional to thevalue of 1 - Sk−1. The parameters α and β (0<α, β<1) represent an individual’ssensitivity to disconfirmatory and confirmatory evidence, respectively. For now, wewill assume they are both equal to 1.

For the sake of argument, suppose that participants’ belief, Sk−1, represents a proposi-tion like “In life or death situations, one should always take whatever means necessary tosave the most lives.” If this is the case, then it is reasonable to think people would endorsethis belief strongly. That is, if we assign it a value in the model, it would be greater than .5.Under these conditions, the Belief-Adjustment model predicts that disconfirmatory evi-dence (the Footbridge dilemma) will have a stronger effect on this belief than equivalentconfirmatory evidence (the Trolley dilemma). As a toy example, suppose Sk−1 takes on avalue of .75. The weight of disconfirmatory evidence is 1×.75=.75, whereas the weight ofconfirmatory evidence is 1×(1−.75)=.25. Suppose further that the Trolley and Footbridgedilemmas provide equivalent evidence for and against the belief, such that the contrast[s(xk)−R] is equal to .5 in both cases. Assuming these values, considering the Trolleydilemma will increase Sk−1 by .125 (Sk=Sk-1+.125), whereas considering the Footbridgedilemma will decrease Sk−1 by .375 (Sk=Sk−1−.375). This is merely an illustration, butwhenever we assume some value for Sk−1 greater than .5, the weight of disconfirmatoryevidence will be greater than the weight of confirmatory evidence (all else being equal).Consequently, if people strongly endorse the beliefs that underlie their judgments aboutthe Footbridge and Trolley dilemmas, then the asymmetry in the Footbridge-Trolleyordering effect could be a direct consequence of how people weigh evidence.2

In the next section of the paper, we provide support for the claim that makingjudgments about the Footbridge and Trolley dilemmas affects people’s moral beliefsto different degrees, that these beliefs are strongly held, and consequently, that theBelief-Adjustment model (1992) can explain the asymmetry in the Footbridge-Trolley ordering effect.

2 Another possible explanation is that some beliefs influence participants’ judgments about the Footbridgedilemma but are not recruited when making a judgment about the Trolley dilemma. For instance, making ajudgment about the Trolley dilemma might recruit a set of beliefs S1, S2, and S3, whereas making ajudgment about the Footbridge dilemma recruits shared beliefs S1, S2, and S3, along with additional beliefsS4, S5, and S6. This means it is possible that consideration of the Footbridge dilemma affects all of thebeliefs recruited when making judgments about the Trolley dilemma, but that consideration of the Trolleydilemma only affects some subset of the beliefs recruited when making judgments about the Footbridgedilemma. If this were right, then people’s judgments about the Trolley dilemma would have a weaker effecton people’s judgments about the Footbridge dilemma than vice versa.

708 Z. Horne et al.

2 Experimental Design

We sought to investigate how making judgments about the Footbridge and Trolleydilemmas affects participants’ moral beliefs and the degree to which they endorse thesemoral beliefs. We conducted two experiments wherein participants made judgmentseither about the Footbridge dilemma, the Trolley dilemma, or a non-moral controldilemma (following a between-subjects design) and then rated the extent to which theyagree with a statement of a moral belief. People’s responses to moral dilemmas are oftendiscussed in terms of “utilitarianism” and “deontology” (e.g., Greene et al. 2001; Prinz2007; Wiegmann and Okan 2012). Accordingly, in Experiment 1 participants rated astatement generally representing a utilitarian viewpoint: “In life or death situations, youshould always take whatever means necessary to save the most lives.” In Experiment 2,participants rated a statement that represented a deontological viewpoint: “You shouldnever kill another person.” Participants’ agreement ratings in control conditions wereused to measure their endorsement of the utilitarian and deontological beliefs statedabove. By comparing the agreement ratings of participants who made moral judgmentsabout the Footbridge and Trolley dilemmas with ratings of participants’ in the controlcondition, we were able to measure the degree to which considering these dilemmasconfirm or disconfirm the beliefs that are recruited when making judgments about theTrolley and Footbridge dilemmas.

We predicted that people would endorse both the utilitarian and deontologicalbeliefs. Additionally, we predicted that the Footbridge dilemma would disconfirm theutilitarian belief more than the Trolley dilemma would confirm it. This result wouldexplain the asymmetry in the Footbridge-Trolley ordering effect. However, it is alsopossible that the Trolley dilemma affects other relevant beliefs more strongly than theFootbridge dilemma, which could undermine this explanation. In Experiment 2 weexamined the effects these dilemmas on the belief "You should never kill anotherperson" to explore this possibility.

3 Experiment 1

3.1 Method

3.1.1 Participants

In Experiment 1, 265 participants (97 female, mean age of 30 years) were recruitedonline, through the Amazon Mechanical Turk (mTurk) work-distributionwebsite. To be eligible for the mTurk posting, workers had to reside in theU.S. and have at least a 95 % approval rate. Participants were compensated$.20 for their participation.

3.1.2 Procedure

This experiment was conducted online, administered using Qualtrics surveysoftware. Participants were required to pass a CAPTCHA test at the beginningof the experiment to eliminate the possibility of automated computer responses.

Belief Updating in Moral Dilemmas 709

After reading instructions, participants first read one dilemma and made ajudgment about the appropriate action. Participants indicated their judgmentabout the moral dilemma using a 6-point Likert scale with the endpointslabeled as “Completely Inappropriate” and “Completely Appropriate.” Then,they rated the extent to which they agreed with a statement of a moral belief.In Experiment 1, this statement was: “In life or death situations, one should alwaystake whatever means necessary to save the most lives”. Participants rated their agree-ment using a 7-point Likert scale with endpoints labeled as “Completely Disagree” and“Completely Agree.” Afterwards, participants answered a pair of simple reading com-prehension questions about the dilemma they read, and also indicated whether they hadever seen or heard the dilemma before. Participants advanced through the experiment attheir own pace, but timing controls ensured they stayed on each page long enough tocompletely read instructions, dilemmas, and questions.

3.2 Results and Discussion

Participants who reported having previously seen the dilemma (52) or who failed thereading comprehension checks (an additional 24) were excluded from analysis. Thisleft 189 participants in the final analysis.

Consistent with our hypothesis, participants in the control condition stronglyendorsed the utilitarian belief statement (mean rating=5.62; SD=1.2). Endorsementwas also high in the Trolley condition (mean=5.26; SD=1.56) and there were nosignificant differences between people’s ratings in the Trolley and control conditions,t(137)=1.657, p=.07. In contrast, participants who read the Footbridge dilemma ratedtheir agreement with the statement significantly lower (mean=4.10; SD=1.52),t(137)=−5.79, p<.001. Moreover, participants in the Footbridge condition gave sig-nificantly lower agreement ratings than participants in the Trolley condition, t(98)=−4.278, p<.001 (see Fig. 1). These results suggest that people construe the Footbridge

3

4

5

6

Control Trolley Footbridge

Agr

eem

ent

Rat

ings

for

Uti

litar

ian

Bel

ief

***

***

*** p < .001

Fig. 1 Participants’ agreement ratings for the utilitarian belief statement across conditions

710 Z. Horne et al.

dilemma as evidence against the utilitarian belief statement, but do not see the Trolleydilemma as evidence in favor of it (in fact, the trend in the Trolley condition wasexactly the opposite).

Participants’ moral belief agreement ratings significantly correlated with theirmoral judgments in both the Footbridge condition (r(48)=.326, p<.05) and in theTrolley condition (r(48)=.459, p<.01), suggesting that this moral belief is importantlyrelated to people’s moral judgments about these dilemmas.

4 Experiment 2

4.1 Method

4.1.1 Participants

In Experiment 2, 202 participants (85 female, mean age=31 years) wererecruited via mTurk using the same criteria and with the same compensationas in Experiment 1.

4.1.2 Procedure

The design and materials of Experiment 2 were identical to those of Experiment 1,with the exception of the moral belief statement. In Experiment 2, participants ratedtheir agreement with the statement: “You should never kill another person”.

4.2 Results and Discussion

As in Experiment 1, participants who reported having previously seen the dilemma(40) or who failed the reading comprehension checks (21) were excluded fromanalysis. This left 141 participants in the final analysis.

On average, participants in all three conditions endorsed the deontological beliefstatement. There were no significant differences between the control (mean=5.08;SD=1.56) and Footbridge conditions (mean=5.43; SD=1.47, t(94)=0.972, p=.33), norbetween the control and Trolley conditions (mean=5.49; SD=1.63, t(102)=1.168,p=.25) (see Fig. 2). Participants’ moral belief agreement ratings correlated onlyweakly (and non-significantly) with their moral judgments in both the Footbridgecondition (r(35)=−.212, p=.21) and in the Trolley condition (r(43)=−.287, p=.06).Somewhat surprisingly, it appears that neither the Footbridge nor the Trolley di-lemmas are taken as evidence for or against the deontological belief statement, andthat this belief may not be as relevant for the formation of judgments about thesedilemmas.

5 General Discussion

We examined the roles that two moral beliefs, one representing a utilitarian perspec-tive and the other representing a deontological perspective, play in producing the

Belief Updating in Moral Dilemmas 711

Footbridge-Trolley ordering effect. We hypothesized that three factors would provideinsight into the asymmetry of the effect: how strongly people endorsed each belief,how their beliefs are related to their moral judgments about the Footbridge andTrolley dilemmas, and how considering these dilemmas affect people’s endorsementof these beliefs.

Although we found that people endorsed both beliefs, only the utilitarian beliefwas significantly related to people’s moral judgments in these two dilemmas. As wepredicted, making a judgment about the Footbridge dilemma reduced participants’endorsements of the utilitarian belief, but making a judgment about the Trolleydilemma had no effect on their endorsements of this belief. Together, our resultsindicate that people recruit the utilitarian belief when making judgments about theFootbridge and Trolley dilemmas, and that this belief is more strongly affected byconsidering the Footbridge dilemma than it is affected by considering the Trolleydilemma. We propose that this pattern of belief recruitment and updating explains theasymmetry in the Footbridge-Trolley ordering effect.

These findings are consistent with the Belief-Adjustment model (Hogarth andEinhorn 1992). If we consider the utilitarian belief as Sk−1, then we should concludethat wFootbridge[s(xFootbridge)−R] is not equal to wTrolley[s(xTrolley)−R]. There are twopossible sources of this inequality: the subjective evaluations of the dilemmas (s(xk))and the weights (wk) assigned to those evaluations. When a to-be-adjusted belief isstrong, as is the case with the utilitarian belief at issue, the Belief-Adjustment modelpredicts that disconfirmatory evidence will be weighted strongly, whereas confirmatoryevidence will be weighted weakly. This is precisely what we observed. Disconfirmatoryevidence—the Footbridge dilemma—had a strong effect on their endorsement of theutilitarian belief, whereas confirmatory evidence—the Trolley dilemma—had no sig-nificant effect. Although it is also possible that the dilemmas are given differentevaluations (after all, they are similar but not identical), the difference in weights issufficient to explain our findings regarding the utilitarian belief, and consequently, theasymmetry in the Footbridge-Trolley ordering effect.

3

4

5

6

Control Trolley Footbridge

Agr

eem

ent

Rat

ings

for

Deo

ntol

ogic

al B

elie

f

Fig. 2 Participants’ agreement ratings for the deontological belief statement across conditions

712 Z. Horne et al.

In Experiment 2, we did not find a significant correlation between people’sendorsement of the deontological belief and their judgments about either theFootbridge or the Trolley dilemmas. Additionally, people’s endorsements of thedeontological belief were not affected by considering either of these dilemmas.Together these results are consistent, as it is unlikely that a belief would be stronglyinfluenced by consideration of a dilemma if the belief were not recruited whenforming judgments about that dilemma.

However, it is unclear why a relevant deontological belief would not be recruitedwhen making judgments about the Footbridge and Trolley dilemmas. It is possible wefailed to reject the null hypothesis when testing the correlation between people’sendorsement of the deontological belief and their judgments about the Trolleydilemma due to insufficient power. Given that we did observe the expected trend inthe data (and the obtained p-value was .06), collecting additional data might revealthat this modest correlation (r(43)=−.287) is statistically reliable.

Supposing this is true, it is somewhat surprising that considering the Trolley dilemmahad no effect on our participants’ endorsement of the deontological belief. At first blush,it seems that people’s judgments about the Trolley dilemma provide evidence against thedeontological belief at issue. And, as people endorse the deontological belief, sequentiallearning models like the Belief-Adjustment model (Hogarth and Einhorn 1992) predictthat disconfirmatory evidence will be weighted strongly.

There are a few explanations for this finding that are consistent with theBelief-Adjustment model. First, it could be that people do not regard theirjudgments about the Trolley dilemma as strong evidence against the deonto-logical belief that we probed. For instance, this could be because people tacitlyallow for exceptions to some strict moral rules. This view, known as ThresholdDeontology, states that deontological rules apply up until a point where theconsequences of continuing to adhere to the rule become too terrible to allow(see Moore ch. 17, 1997; Moore 2012). For example, while torture is normallycategorically prohibited, if torture were the only way to prevent a nuclearweapon from detonating in a major city, torturing would no longer be morallyprohibited. So although participants endorsed the deontological belief, theymay have tacitly held that this rule does not apply beyond some threshold.In this way, their deontological belief would not be undermined by consideringthe Trolley dilemma if the negative consequences of inaction exceeded thethreshold of the deontological rule.

Second, as Hogarth and Einhorn acknowledge in their formulation of the Belief-Adjustment model, some beliefs are more sensitive to new evidence than others. Forexample, people may be largely insensitive to new evidence when updating certain moralbeliefs because they have accumulated overwhelming evidence supporting those beliefs.Perhaps people have accumulatedmore evidence supporting the deontological belief thanthe utilitarian belief, making this belief more resistant to updating in light of newevidence. By choosing appropriate α and β parameter values in the weighting functions,the Belief-Adjustment model can be made to account for the effects of accumulatedevidence. Subsequent investigations could examine whether people have accumulateddifferent amounts of evidence for these two beliefs to justify these parameter values. TheBelief-Adjustment model offers several explanations of our data, although determiningwhether these explanations are borne out requires further empirical research.

Belief Updating in Moral Dilemmas 713

In sum, we examined how people’s moral beliefs influence their judgments about theFootbridge and Trolley dilemmas and how making judgments about these dilemmasaffects their moral beliefs. By using a mathematical model to explain the asymmetry inthe Footbridge-Trolley ordering effect, we also demonstrated that sequential learningmodels can be used to make novel predictions about moral cognition.

Acknowledgments We would like to thank the two anonymous referees at Review of Philosophy andPsychology, Keith Holyoak, John Hummel, Jonathan Waskan, and Tage Rai.

References

Danks, D., T.L. Griffiths, and J.B. Tenenbaum. 2003. Dynamical causal learning. In Advances in neuralinformation processing systems, ed. S. Becker, S. Thrun, and K. Obermayer, 67–74. Cambridge, Mass:The MIT Press.

Greene, J.D., R.B. Sommerville, L.E. Nystrom, J.M. Darley, and J.D. Cohen. 2001. An fMRI investigationof emotional engagement in moral judgment. Science 293(5537): 2105–2108.

Hogarth, R.M., and H.J. Einhorn. 1992. Order effects in belief updating: The belief-adjustment model.Cognitive Psychology 24(1): 1–55.

Liao, S. M., A. Wiegmann, J. Alexander, and G. Vong. 2011. Putting the trolley in order: Experimentalphilosophy and the loop case. Philosophical Psychology :1–11.

Lombrozo, T. 2009. The role of moral commitments in moral judgment. Cognitive Science 33(2): 273–286.Moore, M. 1997. Placing blame: A general theory of the criminal law. Oxford: Oxford University Press.Moore, M. 2012. Ethics in extremis: Targeted killings and the morality of targeted killings. In Targeted

killing: Law and morality in an asymmetrical world, ed. C. Finkelstein, J. Ohlin, and A. Altman.Oxford: Oxford University Press.

Prinz, J. 2007. The emotional construction of morals. Oxford University Press.Rai, T.S., and K.J. Holyoak. 2010. Moral principles or consumer preferences? Alternative framings of the

Trolley problem. Cognitive Science 34(2): 311–321.Rescorla, R.A., and A.R. Wagner. 1972. A theory of Pavlovian conditioning: Variations in the effectiveness

of reinforcement and nonreinforcement. In Classical conditioning II, ed. A.H. Black and W.F. Prokasy,64–99. New York: Appleton.

Schwitzgebel, E., and F. Cushman. 2012. Expertise in moral reasoning? Order effects on moral judgment inprofessional philosophers and non-philosophers. Mind & Language.

Shanks, D. 1985. Forward and backward blocking in human contingency judgment. The Quarterly Journalof Experimental Psychology Section B: Comparative and Physiological Psychology 37(1): 1–21.

Sinnott-Armstrong, W. 2008. Framing moral intuitions. In Moral psychology volume 2 the cognitivescience of morality intuition and diversity, ed. W. Sinnott-Armstrong, pp. 47–76. MIT Press.

Tobia, K. et al. (2012). Moral Intuitions: Are Philosophers Experts?. Philosophical Psychology :1-10.Van Hamme, L.J., and E.A. Wasserman. 1994. Cue competition in causality judgments: The role of

nonpresentation of compound stimulus elements. Learning and Motivation 25: 127–151.Waldmann, M. R., J. Nagel, and A. Wiegmann. 2012. Moral Judgment. In The Oxford handbook of thinking

and reasoning, eds. K. J. Holyoak, R. G. Morrison, 1–68.Wiegmann, A., and Y. Okan. 2012. Order effects in moral judgment: Searching for an Explanation.

Proceedings of the Thirty-Fourth Annual Conference of the Cognitive Science Society. Sapporo, Japan.Wiegmann, A., J. Nagel, and S. Mangold. (2008). Order effects in moral judgment. Philosophical

Psychology (March):2111–2116.

714 Z. Horne et al.