weighted expertise aggregation: an aristotelian middle … · weighted expertise aggregation: an...

Ober: Weighted Expertise Aggregation. 1

Weighted Expertise Aggregation: An Aristotelian middle way for epistemic democracy Josiah Ober. Stanford University. [email protected] Draft of 2011 07 16 Abstract: Epistemic political regimes are concerned with choosing policy options that are right in that in that they track reality better, and thus have better outcomes, than do other options. Analysis of Aristotle’s (Politics 3.11) claim that a large group, using democratic decision-rules, may choose more rightly than an excellent individual or small group yields the approach of Weighted Expertise Aggregation (WEA), a hybrid of deliberation and independent guess aggregation. WEA conjoins diverse kinds of expertise through a time-sensitive process of deliberation and voting. Modeling WEA under conditions of fallible experts and incomplete rules produces a robustly democratic epistemic regime. Although it requires that decision-makers share common knowledge on substantial matters, WEA offers a realistic approach to epistemic-democratic decision-making by long-lived purposeful organizations.

A political regime is epistemic insofar as it is focused on the organization of knowledge (accurate information and true beliefs), rather than simply preferences or interests. Epistemic regimes seek to bring about intended outcomes by making right choices. Choices are right when they track reality better, by taking fuller account of relevant facts about the world, than do other options. Rightness is valued (in this context) because of its consequences: all other things being equal, the right choice is the better choice because it leads to a better outcome. The Callipolis of Plato’s Republic is an example of an ideal epistemic regime, in which philosopher-kings serve as infallible expert rulers. Their right choices, accurately tracking the reality of the Form of the Good, produce and sustain the just and good society intended by its founders. As a political regime, Callipolis is, however, neither realistic nor democratic.

Since Aristotle, political theorists have asked whether a regime can be at once epistemic and democratic, and whether such a regime could be sustained over time in the real world. If, as Plato and other ancient critics contended (Ober 1998), democracy regularly produces worse (because less-right) choices, we must ask whether democracy’s value, as an end in itself (Ober 2007), or as a means to other desired ends (Christiano 2011), justifies worse outcomes. It obviously would be better for democrats if that question were moot. The promise of epistemic democracy is that, under the right conditions, democratic decision-making will result in better choices, so that the independent value of democracy need not be traded off against bad consequences. Were


that promise to be fulfilled, epistemic democracy might offer an attractive alternative both to non-democratic epistemic regimes (e.g. “epistocracy”: Estlund 2003), and to non-epistemic approaches to democracy, including familiar versions of representative and direct democracy.1

If its promise is to be fulfilled, epistemic democracy must address some challenges, including transitivity, collective rationality, and elite control. Right choices, in order to have stable outcomes, must be transitively ordered, such that if A > B and B > C then A > C. Transitivity is not a problem when the choices are made by a rational individual (e.g. a philosopher-king). But democracy assumes multiple decision-makers, and so epistemic democracy must address the cycling problem (i.e. A > B > C > A) that, at least in theory, can lead aggregation-based choice-making into incoherence. It must also show that a decision made through the participation of many individuals can be understood as the rational choice of a collective agent, without reference to metaphysically mysterious conceptions of agency. And, by definition, it must decide matters democratically, avoiding capture by elites or technocrats.2

This paper argues that a system of Weighted Expertise Aggregation (WEA) – counting domain-weighted votes of citizens who are guided in their choices by the reputations of, and reasons given by, experts in multiple domains -- addresses those challenges as well or better than the main contender approaches in the epistemic democracy literature: deliberation (DEL) and independent guess aggregation (IGA). It does so, however, by borrowing essential features from each of the other approaches. WEA is realistic in that it captures some of the ways that right decisions were made in antiquity and are made in the contemporary world. While there are non-democratic versions of WEA (section 3, below), other versions (sections 4 and 5) are democratic in retaining the principle of equal votes and resisting elite capture through agenda control. WEA was, I will argue, the basis for Aristotle’s celebrated claim that, under the right conditions, the many are wiser than any excellent individual or small group. The WEA approach offered here is Aristotelian in that it is inspired by and elaborates upon passages in Aristotle’s Politics and Poetics. It is, however, not fully specified in Aristotle’s texts and might fail to gain his endorsement, whether as an extension of his thought or as a choiceworthy approach to decision-making.

The conditions necessary for WEA are demanding, but realistic. One goal of this paper is to show that right choices are made in the real world by employing WEA, or something quite like it. The necessary conditions for WEA are that the following things are common knowledge: (1) The decision-makers sincerely seek the right answer. (2) The issue on which a choice is to be made has multiple parts, each of which has a specifiable salience. (3) Those parts are explicable as domains of knowledge that can be enlightened by domain-experts who are willing and able to share their expertise. I also assume that (4) the group making the decision exists over time (has a past and a future), and that (5) its members update beliefs about relevant domains and experts, and do so in overall reality-tracking ways. Updating is based (inter alia) on ex post assessment of how specific choices relate to outcomes.

The remainder of this paper proceeds as follows: Section 1 compares the relevant features of weighted expertise aggregation, deliberation, and independent guess aggregation. Section 2 analyzes the famous “wisdom of the many” passage in Aristotle’s Politics (3.11), arguing that it is a compressed account of WEA. Section 3 employs a


passage from Aristotle’s Poetics to fill out a model of WEA under complete rules. In this first model, a right choice among three options is made by aggregating weighted votes of individual experts. This model is not, on the face of it, democratic. Section 4 explains how expanding the model to include mass voting solves the problem, raised by Aristotle, of corruption among experts – and thereby brings WEA into the realm of epistemic democracy. Section 5 introduces a somewhat less stylized model of WEA under incompletely specified rules. The problems and potential of this version are illustrated by reference to decision-making in democratic Athens in the face of an enemy invasion in 480 BCE. Section 6 explains how a community could become better at WEA over time and thereby make better choices more consistently. WEA is not infallible; decision-makers employing WEA process will sometimes make inferior choices leading to bad outcomes. The value of WEA does not lie in always avoiding mistakes, but in learning from mistaken choices as well as from successful ones.

1. Three approaches to epistemic democracy: DEL, IGA, WEA The primary mechanisms discussed in the epistemic democracy literature are

deliberation (DEL), understood as a form of reciprocal reason-giving among citizens (Cohen 1996), and independent-guess aggregation (IGA), understood as a method for mathematically aggregating independent beliefs on matters of fact (Condorcet 1785). DEL and IGA have important real-world applications, for making decisions and for predicting outcomes. Deliberation has been put into practice as a method of polling and for making political decisions (Baiocchi 2005, Fishkin 2009). Examples of aggregating independent guesses include both Condorcet juries and prediction markets applied to political outcomes (Sunstein 2007).

In a spectrum of approaches for organizing diverse information and dispersed knowledge for the purposes of decision-making, DEL and IGA seem to stand at opposite poles: DEL assumes communication among decision makers. It is predicated on exchange of information and reasons, and regards that interaction as necessary for achieving right answers and the best available outcome. DEL values updating by decision makers, on the basis of new information and reasons offered by others, both as a just means to achieving good ends and as an end in itself. IGA assumes independence among decision-makers and regards their independence of judgment as necessary for achieving the right answer. IGA assumes no communication among decision-makers. Updating, based on exchange of views, is regarded as problematically likely to promote cascades and polarization. Pre-decision communication among decision-makers, in ways that violate the independence of their individual judgments, is therefore taken as a source of corruption (List and Pettit 2004). Thus, as in some recent work, e.g. by Cass Sunstein (2007), deliberation and IGA are set up as mutually incompatible. Sunstein’s dichotomous approach suggests that anyone interested in epistemic democracy must make a choice between DEL and IGA based, at least in part, on their differential performance characteristics. Yet, insofar as DEL and IGA each has desirable features, normatively or empirically, the need to choose between them is unfortunate.

Neither DEL nor IGA, in and of itself, fully suits the needs of a democratic community that must regularly make complex, highly consequential, time-sensitive choices. IGA is predicated on agendas (issues and options) that are set exogenously. Lacking the opportunity to deliberate, the voters constituting the group cannot themselves


determine the issues on which choices must be made, or the options from among which they will choose. Nor can they, as a group, establish rules governing issue selection or the option menu. The issues and available options must ordinarily be presented to them by some external authority. Alternatively, in extreme cases, an existential threat (see section 5 for an example) may serve as an exogenously-set agenda, insofar as the group will cease to exist if it does not address the issue. This second sort of exogenous agenda-setting is common to each of the decision-making approaches considered here. Insofar as it sticks by its own premise of complete voter independence, IGA therefore comes with a strong form of external (presumptively elite and technocratic) agenda control built in.

Under DEL issues and options may be set exogenously, by an existential threat or by an external authority. An example of the latter sort of external agenda control is a deliberative poll in which the issue and options are set by the authorities who commission and/or conduct the poll. But a group employing DEL also has the capacity to set its own agenda: It can decide on what issues to take up, and what options to choose among. It can also establish rules governing what issues will be taken up when, and what options will be available on a given issue. Strong forms of DEL require equality of deliberative opportunity and are committed to rules of neutrality. As such, even if it does not require complete consensus (as do some versions of DEL), DEL lacks a mechanism for closure, and thus cannot offer a practical way forward in time-sensitive decision contexts without violating its own premises.3

Real world democracies must be able to set their own agendas (and/or democratically establish rules for doing so) and must make decisions under time constraints. If neither DEL nor IGA, in their pure forms, is fully adequate to the decision-making demands of a real-world democracy, it is worth asking whether the epistemic democracy spectrum can be filled out, by relaxing the premises on which DEL and IGA are predicated. WEA, as a hybrid “third way,” may seem unpromising insofar as a central feature of deliberation (communication of reasons), violates a central feature of IGA (independence of voters). The burdens of this paper are that that WEA, by integrating features from DEL and IGA, realistically enables a group to set its own agenda, to establish its own rules, to make right (and thereby good) time-constrained decisions – and that it does so without losing theoretical coherence. Table 1 summarizes the relevant features of the three approaches.

[Table 1 about here]

In common with other aggregation-based approaches to democratic decision-

making, WEA faces challenges of sustaining transitivity and collective rationality, while avoiding elite control. WEA achieves transitivity and avoids irrationality by relaxing various conditions that together (in theory) render stable transitive choice-making and collective rationality impossible (Arrow 1963; List and Pettit 2011: 42-50). List and Pettit (2011: 53-58) show that collective rationality is possible and cycling is avoided if certain conditions are relaxed; they argue that the most plausible candidates for relaxation are anonymity and systematicity. WEA under complete rules with infallible experts (section 3) relaxes the condition of anonymity, meaning that it violates the condition that votes of individuals must be equally weighted and independent. It also relaxes the condition of systematicity, meaning that it violates the requirement that voting on each part is


independent and that the parts (in List and Pettit’s terms, the premises) are free of interdependence and are treated as having equal salience (i.e. parts are unweighted). WEA under complete rules with mass voting and WEA under incomplete rules relax the condition of systematicity, while retaining a weak version of anonymity: individuals’ votes are equally weighted, but intermixing deliberation with voting compromises the complete independence of individual choices. Versions of WEA under complete rules (sections 3 and 4) are liable to elite agenda control, but, as I argue below (section 5), WEA under incomplete rules is not.

Even if an epistemic approach to democracy can answer the challenges of sustaining transitivity and collective rationality, while avoiding elite control, it faces procedural hurdles: Determining what issues ought to be addressed and when; defining the options among which a choice will be made; choosing rightly among those options under time constraint. The distinguishing features of WEA, which enable it to clear those hurdles, are its assumptions that (1) issues can be parsed into a manageable number of domain-parts; (2) experts in each domain are willing and able to rank options correctly according to the criteria relevant to that domain; and (3) domains of expertise can be weighted according to their salience to the issue at hand. The domains, identity and number of experts, and weighting may be set by rules or may be determined by a combination of deliberation and voting. In either case, the number of domain-parts into which the issue is parsed is (ideally) an optimum that balances two values: Including more domains means that domains can be defined more narrowly; this produces greater clarity in respect to the expertise relevant to each domain and thus aids in identifying relevant experts. On the other hand, restricting the number of domains makes weighting simpler and thereby more transparent.

Sections 3-5 consider three versions of WEA. The first and simplest model assumes that issues, options, and domains are pre-established by institutional rules. It also assumes a single infallible expert in each domain has been identified who will rank the options according to the criteria appropriate to that domain. The second model introduces mass voting into the complete-rules model, as a solution to the problem of corrupted experts. The third model assumes that there are no exogenously given rules for pre-determining issues, options, domains, or for identifying experts. Table 2 sets out the primary features of each version.


2. Aristotle Politics 3.11: The wisdom of the many

In a celebrated passage of the Politics (3.1281a42-b10), Aristotle claims that, under the right conditions, “the many” judge certain matters better than any excellent individual or small group.4 This section seeks to fills out Aristotle’s bare-bones account of the wisdom of the many and to show that the preconditions required for Aristotle’s approach to epistemic democracy are not so demanding as to relegate the wisdom of the many to the realm of ideal theory alone. Aristotle’s highly compressed account of collective right judgment addresses complex decision-making situations, and thus allows us to imagine how diverse expertise might effectively (i.e. in ways that track reality and preserve transitivity and collective rationality) be aggregated by a group of democratic decision-makers confronted with a variety of possible answers.


The relevant passage is laid out schematically, below. Note that the subdivision of the passage into eight sections is my own and will be the basis of subsequent citations of this passage (flagged with §):

1. The many (hoi polloi), of whom none is individually an excellent (spoudaios)

man, nevertheless can, when joined together, be better than those [the excellent few],

2. not as individuals but all together [hôs sumpantas], 3. just as potluck [sumphorêta] dinners can be better than those provided at one

man's expense.5 4. For, there being many, each person possesses a constituent part [morion] of

virtue [aretê] and practical reason [phronêsis], 5. and when they have come together, the multitude [plêthos] is like a single

person (hôsper hena anthrôpon), yet many-footed and many-handed and possessing many sense-capacities [aisthêseis],

6. so it is likewise [like a single person with multiple excellences] as regards to its facets of character [ta êthê] and its intelligence [dianoia].

7. This is why the many [hoi polloi] judge better in regard to musical works and those of the poets,

8. for some judge a particular part [ti morion], while all of them judge the whole [panta de pantes].

(Politics 3.1281a42-b10. Trans. C. Lord, adapted).6 Aristotle’s account is highly compressed. If we are to understand how, in the

concluding phrase (§8), “some judge a particular part, while all of them judge the whole” we will need to do some careful unpacking, in light of information that Aristotle’s original readers would have had readily to hand. Aristotle’s point (§1) is that under the right conditions, a group of ordinary people (hoi polloi) judge better (i.e. make closer-to-right choices with better outcomes) than a few excellent persons. The group achieves its excellence of judgment as a group (§2). Aristotle offers two homey examples to explain how collective judgment can achieve a superior outcome. We may assume that Aristotle expected each example to be familiar to his original readers, and that this presumed familiarity accounts for some of the compression of the passage.

The first example (§3) is an analogy: the “potluck dinner,” to which each prospective diner brings a different gastronomic contribution. The potluck dinner is potentially an excellent whole.7 We must presume, based on Aristotle’s core assumptions about justice as a joint and several common good, that the result of a successful potluck dinner is a better dining experience for each contributor than would be the case if the dinner were provided at one man’s expense.8 The several contributions are the parts that constitute this potentially excellent whole. As such, in order to achieve the dinner’s potential, each of the parts must be of the right sort. The potluck dinner, as a whole, will go wrong if the parts, the various contributions, are not at once diverse and good. Yet how will diversity and goodness of contributions be assured? If we assume independence of contribution choice (i.e. no diner knows ex ante what another will bring), then each diner may just happen to bring the same dish - if we have six diners, we may end up with six courses of pasta salad. Regardless of the goodness of the pasta salad, this will not be


an excellent meal and certainly no better than one provided by an individual. Moreover, potluck dinners are potentially susceptible to free-riding: A free-riding diner might choose strategically to bring something cheap and poor, anticipating that others will bring better fare. If each diner fears being stuck with a sucker’s payoff by free-riders, there will be a race to the bottom and the common dinner will be correspondingly poor.

If the dinner is to be excellent, as Aristotle specifies it can be, the right conditions – the operating assumptions of the contributors to the dinner -- must include some common knowledge and some rules in the form of social norms. First: even assuming that they do not deliberate about specific contributions in advance, the multiple contributors must have good reason to expect particular people to bring a particular sort of food or drink. That is to say, they must share common knowledge of one another so as to be able to predict what each is likely to bring to the table. Next, there must be a norm that ensures that each brings something good to the dinner. These are not excessively demanding conditions. Anecdotally: I have been going on picnics with the same group of friends for the last 25 years. We never prearrange who will bring what, but we do have a good idea of what sort of thing each is likely to bring. Moreover, there is a norm of contributing good things. The result, as visitors to our picnics attest, is that the quality of these potluck meals is high. Per Aristotle’s specification, it is arguably a better experience – gastronomically as well as socially -- than a dinner provided at one individual’s expense. I assume that this sort of predictable outcome is common among long-lived groups with shared norms and stable memberships. So it seems plausible that Aristotle’s original readers had some experience with good potluck dinners.

Aristotle’s potluck dinner analogy, when understood in these terms, gets us some way towards understanding what Aristotle was after in the Politics passage. The potluck analogy introduces the issue of parts (contributions) and wholes (the dinner). The analogy suggests, moreover, that the optimal number of parts falls within a range: with too few dishes the meal will be excessively plain. Yet at some point it will not be improved, indeed may be made worse, by the addition of further dishes. The analogy also shows why the members of the group must share some relevant forms of common knowledge, and why they must share social norms about quality.

Aristotle’s second example (§7) concerns judgment of a musical/poetic production: He states as a matter of fact that hoi polloi do judge better (than the few) in regard to “musical works and those of the poets.” Aristotle may have various musical/poetic productions in mind here; recent work by classicists has demonstrated the richness and diversity of the culture of musical performance in the classical Greek world (Wilson 2000). But it seems certain that among the productions Aristotle and his original readers had in mind was the performance of drama, and especially tragedy. Tragic theater was, in Aristotle’s view and that of modern scholars, the definitive “musical/poetic” venue in classical Athens, where Aristotle lived for most of his working life. It is a safe guess that Aristotle’s original readers were reasonably familiar with the rules governing the judgment of dramatic performances.

In classical Athens, prior to a Dionysian festival, three tragic poets (and three comic poets) were chosen by a state official to present works for performance. Each chosen tragic poet presented a group of three plays. After the performances, the three play-groups were judged; on the basis of that judgment the tragic poets were awarded first, second, and third prizes. The judging was by a panel chosen by lottery from among


the citizens of Athens. In practice, the mass audience of several thousand spectators, by its response to the performances, gave the judges their cue - as appeals for the audience’s good will in each of Aristophanes’ extant comedies clearly demonstrate.9 It is, therefore, a likely hypothesis that Aristotle had the mass judgment of a theater audience in mind when he stated that hoi polloi judge better in regard to musical and poetic works.

In §4 of the Politics passage Aristotle specifies that each member of the decision-making group possesses a constituent part of virtue and practical reason. These parts can be effectively aggregated: At §5 and §6, Aristotle states that a mass can, under the right conditions, be like a single person -- yet unlike a single person it possesses a multitude of sense-capacities, and likewise the facets of its character, and its intelligence are multiplied. Aristotle seems to mean that the group possesses among its membership multiple and diverse conjunctions of sensibility, character (virtue), and intelligence (practical reason), and that these conjunctions are relevant to collective judgment. Individuals manifesting these multiple conjunctions are, therefore, explicable as parts (§4) that, when properly aggregated (come together), produce a right single-person-like decision. Since the example (§7) is judgment of musical/poetic productions, and since (so I have argued), is Aristotle had tragedy in mind, some of the multiple conjunctions may be understood as forms of expertise that bear on the judgment of the multiple parts that make up a good tragedy (§8).

Aristotle’s point here seems to be that a few talented individuals may indeed have the sensibilities, character, and intelligence to be a very good at judgment of a given part of tragedy. But the excellent few will lack some conjunctions of sensibility, character, and intelligence – some forms of expertise -- relevant to judging the whole. A large enough group, with the right sort of diversity, will possesses all forms of expertise necessary to making a right decision. The group will judge well if it is able to bring the relevant forms of expertise to bear on the constituent parts of the tragedies being judged, while giving each form of expertise the right weight. If the account is to avoid metaphysical mysteriousness, the aggregation process must be accomplished without losing sight of the fact that, while it is in some ways like a single person, the group actually consists of individuals. The aggregation is explained unmysteriously if we suppose (as the context of the passage clearly implies) that Aristotle was referring to a decision-making institution. If we assume that Aristotle was in fact referring to the judgment of tragedy in §7, we can fill in his account of the “wisdom of the crowd” case by reference to actual institutional rules, domains of expertise, and weighting. Analyzing how a group of the right sort, acting as a quasi-person through procedural rules, could correctly rank tragedies, allows us to put some flesh on the bare bones of Aristotle’s account of the wisdom of the many.

3. WEA under complete rules with diverse and infallible experts

Athenian institutions stipulated in considerable detail the rules for the awarding of prizes for tragic poets. The choice was among three poets, who had been selected in advance by the relevant Athenian magistrate. The ranking (first, second, third prize) was based on the performance of three tragedies by each poet. In the Poetics, Aristotle provides his readers with an account of the six parts (merê) of tragedy that (for our present purposes) we may regard as additional rules. By specifying the relevant parts that make up the whole that is a good tragedy the Aristotelian rules point to relevant domains


of expertise. Aristotle’s rules also give us guidance on the correct weighting of each part (plot, character, diction, thought, spectacle, and song):

Necessarily then every tragedy has six constituent parts [merê], and on these its quality depends. These are plot [muthos], character [êthê], diction [lexis], thought [dianoia], spectacle [opsis], and song [melipoiia]. Two of these [diction, song] are the means of representation: one [spectacle] is the manner: three [plot, character, thought] are the objects represented. This list is exhaustive, and practically all the poets employ these elements, for every drama includes alike spectacle and character and plot and diction and song and thought. (Poetics 1450a6-14) The Poetics passage recalls the Politics passage; most obviously, both are

concerned with character and thought as natural causes and co-determinants of the quality of an action.10 Moreover, both passages are concerned with how subsidiary parts (merê, moria: these terms are used interchangeably by Aristotle) constitute a whole. With reference to the potluck analogy, we can think of each of the six parts of tragedy as analogous to gastronomic contributions that make up a fine dinner, although the analogy is inexact in that it is judgment of quality that is Aristotle’s concern in §7 and §8 of the Politics passage.

In order for a judgment of a tragedy group to be “best,” each of the six constituent parts must be taken into account and properly weighted; ignoring or improperly weighting any relevant part will result in a flawed overall judgment and thus in a bad outcome: the prizes will be awarded to the wrong poets. Aristotle specifies that the six parts are ranked in the following order of importance: plot, character, thought, diction, song, spectacle (Poetics 1450a-b). Thus, plot must be most heavily weighted in the judgment, spectacle least heavily, with the others in between. The exact weights remain unspecified. In the model that follows (Table 3) the weighting of the six parts (Pw) is 8, 6, 5, ,4, 3, 2; thus the most important part (plot = 8) is assumed to be four times the least important (spectacle = 2). There is no way to guess how close this is to Aristotle’s own weighting, and of course changing the weights produces different outcomes. The following simple (no interaction terms) and stylized model is meant only to show how the tragedy ranking problem is solved, when we assign specific weights to the six parts, and (extrapolating from Aristotle) assume some things about institutions and common knowledge.


In this stylized case, the assumed features of the tragedy choice are as follows:

1. The number of options (three tragedy-groups) and the actual options (T1, T2, T3) are set by the rules of the Athenian institutions governing the festival.

2. The six parts of tragedy are common knowledge (i.e. Aristotle’s analysis of the parts of tragedy is known and accepted by all).

3. Each part is a domain in which there is a known expert who is capable of judging correctly (i.e. who will rank the three options accurately, in terms of the part).

4. Correct weighting among the parts is common knowledge (i.e. Aristotle’s hierarchy, now assigned particular weights, is known by all).


The procedure assumed in the stylized model is as follows: Judgment of each part is by an independent expert. Having watched the performances, the expert correctly ranks the relevant part of each tragedy group. For example, the Plot expert ranks T1 highest, yielding an unweighted vote of 3, and a weighted vote of 24. After each expert has voted, the weighted aggregate for each tragedy group is tallied, and the ranking is determined by the numerical scores. The outcome (awarding of prizes) is determined by the ranking. The procedure ranks the three options transitively and shows how closely they were ranked. The highest possible weighted aggregate for a tragedy-group is 84 (i.e. all six experts rate that group best on all six elements); the lowest possible ranking is 28; thus not only the fact, but the magnitude of a given poet’s victory is determined by this decision method. As a result of the aggregation of weighted expertise, T1 (59) is awarded first prize, beating T2 (56) by three votes; T2 is in turn three votes ahead of T3 (53).11

Aggregating domain-weighted votes of experts yields the correct (Aristotelian) ranking of the tragedy groups in the order 1, 2, 3. This apparatus would be otiose if the same ranking could reliably be achieved more simply, e.g. without resort to weighting or by consulting fewer experts. But the correct result produced by WEA is not, in the weighting scheme employed here, achieved by any of the obvious short-cuts sketched in alternative ranking plans.

• Plan 1: Judgment by expert of most important single part decides. Plot is the most important part: 1 3 2. Fail

• Plan 2: Expert egalitarianism. Six judges, each with equal vote (unweighted ratings); result = unweighted aggregate: 2 1 3. Fail.12

• Plan 3: Coalition of experts of the two most important elements decide, weighted ratings. Plot and character decide: 3 1 2. Fail.13

o Note than other two two-member coalitions also fail. Note that unweighted two-member coalitions either fail or produce three-way tie.

• Plan 4: Coalition of experts of three most important elements decide, weighted scores. Plot and character and thought decide: 3 1 2. Fail.14

o Note that a three-member “unimportant elements” coalition also fails. Note that unweighted 3-member coalitions produce a three-way tie.

The epistemic value of WEA is demonstrated by the failure of simpler (no

weighting, fewer decision-makers) ranking plans. It is also intuitive in that informal real-world examples of something like this version of WEA are readily at hand in the decision-making processes of individuals who are faced with making choices on matters that are divisible into domains of expertise. Suppose, for example, that an experienced opera critic must write a review that includes a rating (say, one to four stars) and a ranking (better or worse than other operas this season). Between acts of the performance she seeks the opinion of audience-members thought to be expert in the various aspects of the performance she will highlight in her review (singing, staging, acting and so on). Her experience helps her to identify the right experts and to give the right weight to the various parts as she composes her review and decides on her rating and ranking.

Or suppose that an attending physician must decide among several treatment options for a patient. The physician’s experience with this sort of case leads her to recognize that the case has several distinct aspects, and that each aspect demands a


certain expertise. The attending physician consults other medical experts, each of whom recommends one option or another, based on his or her specific domain of expertise (e.g. surgery, diet, psychology). Under the right conditions, the attending physician aggregates appropriately weighted expert opinions, and thereby recommends a best option for the patient’s treatment.15

Neither the opera critic nor the attending physician is likely to assign numerical values to the weighted expert opinions that contribute to the final decision, but the basic features remain: complex matters that can be broken down into parts, known experts for each part, options delimited by rules, weighting. In each case the decision must be made in a timely way (the critic has a deadline, the treatment must begin before the patient dies). The opera critic and attending physician examples show that this first version of WEA enables a group to be quite “like a single person” in coming to a decision. It is not, however a democratic approach in that the agenda not set by the decision-making group (the experts), voting in each domain is by a single dictator-like expert, and the experts’ votes on ranking are unequal (due to weighting). Moreover, as we will see (section 5), this undemocratic cloistered expert approach (Ober 2008: 1-3) is ill-suited to addressing complex situations that lie outside the group’s ordinary experience.

4. WEA under complete rules with mass voting In the opera critic and attending physician examples considered above, the provision of experts in several domains is essential to achieving the superior outcome, but there is no need for a mass of voters. The rules of the situation determine the range of options; after the experts have weighed in, all that is needed is for a competent authority to count the experts’ weighted votes and to announce the result. Yet Aristotle’s “wisdom of the many” passage assumes it is a “multitude,” acting like a single person (§5), that judges the whole; the superior judgment on musical and poetic works is made by “the many” (§7) -- not simply by a properly diverse body of experts. If WEA is to be regarded as a form of epistemic democracy, as opposed to a defensible form of technocratic decision-making, we will need to take Aristotle’s presumption seriously: Under what conditions will a mass of voters improve the outcome?

The “judging tragedy” case benefits from adding a stage of mass judgment of expert opinion when we relax the assumption that each expert always votes correctly. In the real world experts are sometimes wrong – their data may be faulty or their methods flawed. But even if we assume that an expert always knows the right answer, we may not assume that the she will always express her true belief. A self-interested judge may misrepresent her honest judgment because she perceives some benefit in so doing. Absent the right incentives (section 6) we would expect strategic behavior from rationally self-interested experts, and such behavior could obviously corrupt the process described in the previous section. If one or more of the tragedy-judging experts is corrupt in Aristotle’s sense (he values a private benefit above honesty and the common good), the likely result, based on Table 3, will be the wrong outcome in the awarding of prizes. Aristotle was very aware of the possibility of corruption in decision-making. In his discussion of the possibility of mass wisdom he explicitly states that among the advantages of decisions made by a crowd is that such decisions are less subject to corruption: Aristotle uses the metaphor of a large quantity of water that is relatively unpolluted by a small admixture of impurity (Politics 3.1286a3-35).


The Aristotelian solution to the problem of corruption is for a mass of voters to choose among rankings proposed by several experts for each part. Schematically: after each domain-expert has explained to the mass the reasoning behind his proposed ranking (“some judge a particular part”: §8), many citizens cast their equally-weighted votes for a ranking of tragedy-groups in the relevant domain. Each voter will choose based on the reputations of the experts and/or the cogency of their reasons. This vote of “all together” (§2) stands in for a judgment of a single infallible domain-expert: The mass is thus non-mysteriously acting “like a single person” (§5). Insofar as the mass judges well, based on its aggregate assessment of reputations and reasons, its collective decision is “better than [that of the excellent few]” (§1), even if we assume that the few possess all relevant forms of expertise, because the corrupt expert’s mis-ranking is rejected and so does not corrupt the outcome. The voting continues for each part – “all of them judge the whole” (§8). The votes on the parts are then weighted (based on the rule), counted, and the final ranking is announced. Although the domains are weighted, because each voter casts a vote for each part, each voter has equal say in the overall outcome.16

The process requires that at least some experts are competent at explaining the reasoning behind their rankings to non-experts, and that they value their reputations. It also requires that the mass be able to judge well, based on reputations and reasons. Aristotle has specified (§1) that the crowd is not made up of individually excellent people – that is people with high levels of “complete” virtue and practical wisdom – i.e. the mass does not consist of the sort of superior persons depicted as ideal moral subjects in the Nicomachean Ethics. As we have seen, each individual possesses only a “part” (morion) of virtue and practical wisdom (§4). Key to the group’s capacity to judge supremely well – i.e. to act as a qualified (if not infallible) quasi-person (§5) -- is its collective possession of an adequate level of virtue/character and practical wisdom/intelligence.

In the discussion that follows the “wisdom of the many” passage, Aristotle emphasizes that it is only when the decision-making group is of the right sort that it will make superior judgments (Politics 3.1281b15-20, 1286a36-38). Being of the right sort means being collectively attentive the common good, relatively uncorrupted, and marginally competent at making certain kinds of social judgment. In reference to Condorcet’s (1785) jury theorem, we may say that average individual competence in the relevant sort of social judgment is over 0.5, on a scale of 0-1. The voting body may include incompetent and corrupted voters, but the mass is assumed to be large enough to dilute their influence; it is not systematically corrupted or dominated by incompetents. The group must also (per the implications of the potluck analogy) be collectively sensible to the loci of relevant kinds of expertise among its membership, and it must pay proper attention to the reputations and reasons given by those who are especially adept in each domain.

Aristotelian WEA arrives at right answers by a path that is in some ways similar to, but in other ways quite different from IGA. Like Condorcet (1785), Aristotle assumes a group of preference-sharing decision-makers, each of whom believes that there is a best choice and sincerely seeks that best choice. Like Condorcet, it is the marginally positive competence of the majority that is aggregated at scale to achieve a reliably correct result. Aristotle did not do Condorcet’s math, but his intuition seems to point to something like the jury theorem. Yet unlike Condorcet, in Aristotle’s approach (as reconstructed here) a central role is played by reason-giving and intra-group diversity. Diversity in expertise


(Page 2007), along with the aggregation of marginal correctness, is what grants a WEA group its potentially superior capacity for judgment. Aristotle may therefore assume that, were they to decide on the issue independently, as fallible individuals inattentive to the experts among them, most members of the decision-making group would choose wrongly. That is, the group would come to the wrong conclusion if it were to decide the issue as a Condorcet jury without the benefit of parts, experts, and weighting.

Aristotelian WEA replaces Condorcet’s assumption of individual marginal likelihood of correctness on the matter being decided with an assumption that some people are expert in any given domain and that a majority of voters will be reasonably good at judging experts. The assumption of “marginal likelihood of correctness” remains an essential feature of the process, but it is deferred: away from the substance of the matter at issue and towards a sort of social knowledge. The “character and mind” (§6) of the single-person-like collectivity is, on this reconstruction, plausibly capable of making a correct judgment on the character and reasoning of domain-experts. The collectivity comes to the right answer on an issue with several parts, but the voters need not be domain-experts nor need they make individually correct judgments on the complex whole. As in the “infallible experts” version of WEA considered above, “mass decision/Aristotelian WEA” is intuitive in that informal examples are readily available in the real world. Suppose, for example, that the faculty of a large and well functioning (thus sincerely best-choice-seeking) academic department must decide among three candidates for a new faculty appointment. The various parts constituting the issue of choosing a best candidate (say: research, teaching, service; or theory-building and empirical theory-testing), and the relative salience of each part, are likely to be fairly well specified by rules, formal and informal. There are likely to be certain individuals recognized as especially expert who will be consulted on each of the several parts (writers of external letters, highly regarded insiders). There may be concern about the quality or impartiality of some experts. Consistency of judgment among experts, the quality of reasons given by the experts, and the effects of reputation are all likely to play a part in the decision of each of those casting a vote. In the end, the members of the department vote, and (if the process has worked as it ought), the vote chooses the best candidate based on the weighted criteria and expert testimony. The collective memory of the department will be likely to record the correlation between process and outcome (did the chosen candidate turn out to be excellent?); the reputations of experts are likely to be updated accordingly. This second version of WEA is democratic in that it employs mass voting and equal votes. It allows for a timely decisions. Yet insofar as the rules governing options and domains are set by an authority other than the group, it remains liable to elite control.

5. WEA under incomplete rules The decision-making models sketched in sections 3 and 4 are meant to capture key

features of real decisions. Yet in the real world, rules governing the relevant domains of expertise and the relative weighting of multiple domains are never truly complete. In the examples of the opera critic, attending physician, and faculty vote, rules guide decision-making, but the rules nonetheless under-determine both domains of expertise and weighting – as they certainly did in the assigning of prizes to tragic poets in classical


Athens. The prior discussion of stylized complete-rules models was meant to highlight the role of rules in decision-making. It also points to what is needed if rules do not specify the issue to be decided, the number of options to be considered, the domains of expertise that are relevant to the issue, or the weighting among domains. Under those conditions, WEA may escape the shadow of agenda control and thus become fully democratic. Yet rules need not remain radically unspecified for fully democratic WEA. Decisions made under something approaching complete rules may rightly be regarded as democratic if the rules specifying options, domains, and weights were democratically established. Like DEL and unlike IGA (section 2), WEA has the potential to establish its own agenda-setting rules. Less stylized complete-rule versions of WEA can thus be regarded as parts of a democratic order, so long as that order remains grounded in decision-making under incomplete rules.

In the historical case developed below, a democratically appointed deliberative/voting council prepares an agenda for a larger deliberative/voting assembly empowered to make binding decisions on the most important matters. Incomplete rules need not mean anarchy: In this case, democratically-established rules govern some aspects of deliberation and voting. The background WEA conditions, listed in the introduction, are presumed to hold: the group exists over time, seeks best choices, and updates; issues are divisible into parts, each with a specifiable salience and identifiable domain-experts; all of this is common knowledge. The case describes a situation in which rules of thumb on domains and weighting are bootstrapped in the course of the decision process. I argue below (section 6) that an epistemic democracy can, over time, democratically enact rules that will go some way towards specifying domains, weighting, and other elements in the process.

The historical case developed here -- the response of democratic Athens to the threat of Persian invasion in 480 BCE, as described by the historian Herodotus (Histories 7.140-44) -- was well-known to Aristotle. Like the passages from Aristotle in sections 2 and 3, Herodotus’ account is compressed, but may be interpreted as describing a version of WEA. Much of the following narrative remains hypothetical. It is meant to illustrate a decision-making process parsed into discrete stages. The real process was surely messier, but none of the stages is counter-intuitive or contradicts known historical facts. The process described below is within the bounds of what historians of antiquity have been able to reconstruct about decision-making procedures in the Athenian council and assembly (Rhodes 1985, Hansen 1987, Missiou 2011).

In 480 BCE, as a massive Persian force advanced west into Europe, the Athenian state was confronted by a clear and present danger. The issue of how to respond to the invasion would ultimately be decided at a meeting of the citizen assembly, open to all adult male Athenians and attended by several thousand of them. The agenda for that (as every) assembly meeting was set by a democratic council. The council was a broadly representative body of 500 men, chosen by lot from Athenians over age 30. The council’s role was to define policy options. The Greek name for the council (boule) refers directly to its deliberative function (bouleuein = “to deliberate”), but it is quite certain that votes were also taken. We have no record of the debate in the council or of its recommendation on the invasion issue. We know that by the time of the assembly meeting three primary options – flee, fight at sea, fight on land – had emerged, almost certainly in the course of the council’s deliberations. Because the agenda for the assembly was announced in


advance, ordinary citizens had the opportunity to deliberate among themselves before the decisive meeting, at which a majority vote of the citizenry would decide how the state would respond to an existential threat.

Although Athens was used to fighting its Greek neighbors, invasion by the Persian empire was an extraordinary issue. Because Athenian decision rules did not (indeed could not) specify parts or weighting for exceptional issues, and because the decisive assembly vote would be on the issue itself (rather than its parts), each citizen had to take a share of the collective responsibility for each of four elements that were pre-determined by the mass-voting/complete-rules WEA model.

1. Specification of the primary domain-parts of the issue 2. Identification of experts in each domain 3. Weighting of the several domains 4. Calculation of aggregate weights to determine the conclusion.

There would have been substantial variance among the citizenry (councilmen and those in the assembly) in respect to capability for each of these tasks. Table 4 suggests a hypothetical distribution of capabilities among a sample population of 25 citizens. A substantial minority of the citizens in the hypothetical sample falls below the presumed point (0.5 on a scale of 0-1) at which he will make a positive contribution to the relevant element. Yet the mean competence level is above 0.5 in each element. Thus we can appeal to the aggregation of positive marginal likelihood of correctness, familiar from Condorcet’s jury theorem, to suggest that the collectivity will do adequately well on each element. As the numbers increase (500 councilman, thousands of assemblymen) the marginal likelihood of correctness grows. The aggregation of marginal competence will, in this case (as in other forms of WEA), be augmented by deliberations by which the council sets its own agenda and that of the assembly.

[Table 4 about here] The council takes up the issue of the Persian invasion in a series of meetings. These are time-constrained but substantially less so than the subsequent decisive meeting of the assembly. The deliberation begins by determining the relevant domain-parts. Based on their previous meetings, at which they deliberated other issues, the councilmen have a reasonably good sense of the distribution of capabilities among their membership in respect to recognizing parts, setting weights, and identifying and assessing experts – that is, they have got something like Table 4 in their heads.17 Because they are assumed to be sincerely best-choice seeking and the stakes are very high, the councilmen seek to make use of this distribution of expertise by allowing those who are thought most capable in each area to take a lead in the deliberation. Councilman A makes a strong argument for the general importance of the attitude of the gods (and thus the morale of the populace).18 In the course of deliberations, Councilman A’s proposal for the salience of this part gains majority support (in Table 4: at least the 13 of the 25 voters with a “recognize parts” capability of 0.5 or above are assumed to vote in favor) and thus it becomes one of the domains in which expert advice will be sought. Councilman B, C, D, Q subsequently make similarly cogent and successful arguments on behalf of Persian plans and capacities, strategic advantages potentially gained by local knowledge, Athenian logistical capacities, attitudes and capacities of Athenian allies. The addition of new


domain-parts ends at point at which there is no majority in favor of another part’s high salience to the issue.

In the course of the council’s deliberations, some agreement also emerges on the right weighting of each part. Given the hypothetical distribution of capabilities in table 4, in which capability in identification of parts is not strongly correlated with skill at weighting, different councilmen (E, G H, I, X) take the lead in advocating the proper weighting. As domains and weights are worked out, experts are identified (on the motions made by B, C, D, V, Y). The experts address the council and their recommendations are assessed by all present, although the responses of C, E, F, Q, Y are especially influential. As in the identification of parts, deliberation ends when there is no majority in favor of continued discussion of weighting or further expert testimony. In the course of this iterated process of deliberation and voting, the three primary options (flee, fight at sea, fight on land) emerge and are tested against the views of domain-experts in the ongoing deliberations. The agenda is duly set, and the assembly is called. The results of the council’s deliberative/voting process are reported: in this case, we can suppose that the report details the three options (with some discussion of associated domains, weights, and experts) rather than making a specific policy recommendation. The council’s bootstrapped rule-like specifications are taken seriously by the assembled citizens, because time constraints are now stringent: The assembly must make a decision by day’s end. Proposals from the floor might nonetheless have led to changes in options, domains, and experts under consideration. Anyone that the assembly was willing to accept as an expert in a relevant domain was entitled to speak to the issue. Those regarded as non-expert in the relevant domains would quickly be shouted down (Plato, Protagoras 319b-c); the time constraints did not allow the luxury of irrelevant comments. Yet Herodotus reports that “many opinions” were offered; clearly there was a range of expert testimony for and against each of the three primary options.

The winning option in a successful incomplete-rules WEA process will be the one that best takes into account the most heavily-weighted domains and the most credible experts. In this case, Themistocles’ proposal to fight at sea carried the day. Because the vote was conducted by estimating the number of raised hands, rather than by counting ballots, the vote was observed by the assembly as a “unified whole” (Schartzberg 2010). The assembly’s vote in favor of the fight-at-sea option was a direct expression of the demos’ collective choice and was immediately put into action. Themistocles’ reputation as a trustworthy leader and as an expert in naval affairs certainly played a role in the choice to fight at sea. Yet Herodotus makes it clear that Themistocles’ ability to offer reasons that addressed each heavily-weighted domain of expertise (prominently including the attitude of the gods) powerfully influenced the final vote.19

Given the impossibility of specifying all counter-factuals, we cannot know whether the assembly chose the best option. Yet appears that the mass of Athenian voters proved capable, as a collectivity, of accurately aggregating the weighted domain-parts. The decision, as Herodotus was at pains to point out, changed the course of Greek history in ways that were overall positive for the Athenians. Herodotus states (7.139) that the Athenian decision to fight at sea determined the outcome of the war; after winning the sea-fight, democratic Athens went on to become the preeminent state of the Greek world. Herodotus also pointed out (5.97.2-3) that decisions made by the Athenian assembly were


not always right and sometimes had bad consequences. Yet the Athenians’ capacity to organize useful knowledge through democratic processes, and thereby to make many right choices on issues of public policy, helped to produce Athens’ record as an exceptionally influential, secure, and prosperous city-state over the next five generations (Ober 2008).

6. Conclusions: WEA over time

The Persian invasion example in section 5 defines an outer limit of WEA decision-making in that the issue (massive land-sea invasion by the Persian empire) that must be decided is one that none of the participants had faced before. The Athenians did not regard the extraordinary event as a justification for suspension of democratic decision-making process.20 Yet due to the exceptionality of the issue, the options, domains, and weighting all had to be determined ab initio by the citizenry, through the deliberative/voting process itself. Many decisions made by long-lived organizations concern recurring issues (in Athens: festivals, taxes, public buildings etc.). Each time an issue recurs, there will be some contextual specifics that must be taken into account. Yet collective experience, framed over time by institutional rules, will provide guidance on options, domains (and their associated experts), and weighting. To be effective over time, an epistemic approach to democracy must find an appropriate balance between deciding on the basis of experience and rules, or alternatively on the basis of options, domains, and weightings that arise afresh from the deliberative/voting process. This may be understood as variant of a challenge faced by all complex, long-lived organizations: how to balance the value of routinized learning against the value of experimental innovation (Levitt and March 1988).

A community employing a WEA approach to decision-making may improve its epistemic performance over time (i.e. make right choices more consistently) if individual participants are honest when offering expert testimony and if they improve their performance (i.e. score higher on the 0-1 competence scale) in recognizing and weighing parts, and identifying and assessing experts. Participants will be more likely to be honest and to assume the costs associated with seeking improvement if they are offered the right reasons. Those reasons might be in the form of material incentives and sanctions (rewards or punishments), or persuasive cultural narratives about “why we do things that way (speak honestly, seek improvement), around here.” Learning by individuals may be in part formalized (training, mentoring), and may be tested by the sort of ex post performance assessments common among contemporary “learning organizations” (Davenport and Prusak 1998, Dixon 2000, Garvin 2000). To the extent that the several elements of the WEA process can be learned, we would therefore expect organizations offering the right incentives and possessing the right culture (in terms of common knowledge and shared values) to achieve better aggregate performance over time.

Learning may also be in the form of self-knowledge and tacit social knowledge. As, with increasing experience in deliberation and voting, individuals come to recognize their own strengths and limits (i.e. where they stand in the distribution modeled in Table 4), and the strengths and limitations of their fellows, habits are formed and reputations established. To the extent that citizens learn to assert themselves on matters in which they perform better than the mean, and to attend to more-capable others in matters on which


they themselves fall below the mean, we would once again expect marginal aggregate improvement over time.

Finally, a democratic organization may itself become wiser over time by instituting rules that codify option-choice and weighting and thereby make optimal use of expertise. Excessive codification risks ossification and/or elite control. But democratic constitution-making may, in principle, find a ground in which the rules are well enough specified to capture most relevant expertise and collective social knowledge, while remaining sufficiently porous to allow for innovation through the iterated process of deliberation and voting. All of this is, once again, meant to be intuitive and examples are readily available in the history of states and other purposeful organizations (Manville and Ober 2003; Ober 2008). To the extent that the potential for epistemic learning is realized, it will produce a democratic advantage. That advantage arises (if and when it does) for reasons somewhat different from the political-economic factors on which the recent literature addressing “democracy and state performance” has primarily focused.21

The optimistic WEA-as-civic-education story remains, however, only one possibility. Consistent improvement, without setbacks, is unlikely to be (or ever to have been) realized over the entire history of any long-lived democratic organization. An organization practicing WEA is not proof against being misled by experts with good reputations who (by mistake or strategic calculation) propose attractive but ultimately unsound reasons in support of inferior options. Nor is it immune to deliberative pathologies, including cascading and polarization. Wrong decisions will be made if the mass of voters are, in the aggregate, marginally wrong (below 0.5 on the 0-1 scale) rather than, as in a successful Condorcet jury, marginally right (above 0.5). The same Athenian democratic process that chose the right option when faced with Persian invasion in 480 BCE, chose the wrong option in deciding to launch a massive invasion of Sicily in 415 BCE, in the midst of the Peloponnesian War. As in 480, there were detailed deliberations and a series of votes; various options were proposed, and experts with well-established reputations offered detailed reasons. Yet in 415 there was a cascade of enthusiasm for an invasion plan predicated on misinformation, and the cascade swamped other options. The outcome was a reversal of Athenian fortunes, leading to Athens’ temporary eclipse as a major Hellenic power (Thucydides books 6 and 7).

Bad decision-making, arising from what he saw as a flawed democratic process, was Thucydides’ explanation for Athens’ failure in the Peloponnesian War (Ober 1998, ch. 2). Whether or not that is the right explanation for Athenian state performance, it is worth noting that Thucydides’ argument was predicated on assessing democracy as an epistemic process, aimed at making right choices. Thucydides in effect demands that a proponent of WEA explain why pathologies leading to wrong choices arise within epistemic-democratic procedures and how they might be prevented. If the arguments offered here go through, that challenge will need to be addressed in future work. If, on the other hand, democracy is properly understood as nothing more than a system for aggregating preferences, it is hard to see how the process of democratic decision-making could, in and of itself, explain better or worse outcomes. The tendency to regard democracy as a system of preference aggregation is presumably one reason that modern social scientists seeking to explain correlations between democratic institutions and state performance have paid scant attention to democracy’s epistemic features.22


The hope offered by WEA, as an approach to epistemic democracy, is not that mistakes will always be avoided. It is rather that, over time, mistakes will become part of a learning/updating process – as they did in Athens, where decision-making procedures were revised in ways that enabled Athens to avoid the cascades that led to the poor choice of 415 BCE (Ober 2008: 64-69). If the right lessons are learned from both successes and failures, and if these lessons are captured by experience and rules, then, over time, generally better options will be chosen, resulting in generally better outcomes. Whether this hope is justified is not a question that classical or contemporary political theory can readily answer. Nor can it be answered by reference to any given historical case. More and deeper empirical studies, experimental as well as historical, are clearly needed.

Such studies might focus on how WEA with complete rules and infallible experts models relatively well- or poorly-functioning committees, boards, scholarly and creative collaborations, and other group agents whose purposes depend on identifying and drawing from diverse domains of expertise. WEA with complete rules and mass voting may be at least partially exemplified by some versions of deliberative polls (Fishkin 2009). WEA under incomplete rules may be taken as modeling the work of constitutional assemblies whose work must be ratified (and perhaps amended) by a popular vote. If, in short, I am right in claiming that features of WEA have long been prevalent in organizations of various kinds, then there is reason to think that future empirical work will determine whether or not WEA deserves to be greeted as good news by democrats.


BIBLIOGRAPHY Acemoglu, D., & Robinson, J. A. (2006). Economic origins of dictatorship and democracy. New York ; Cambridge: Cambridge University Press. Anderson, E. (2006). The Epistemology of Democracy. Episteme: Journal of Social Epistemology, 3(1), 8-22. Aristotle, Robinson, R., & Keyt, D. (1995). Politics III and IV. Oxford and New York: Oxford University Press. Arrow, K. J. (1963). Social choice and individual values (2d ed.). New Haven: Yale University Press. Baiocchi, G. (2005). Militants and citizens: The politics of participatory democracy in Porto Alegre. Stanford, Calif.: Stanford University Press. Boix, C. (2003). Democracy and redistribution. Cambridge: Cambridge University Press. Cohen, J. (1986). An epistemic conception of democracy. Ethics, 97(1), 26-38. Cohen, J. (1996). Procedure and Substance in Deliberative Democracy. In S. Benhabib (Ed.), Democracy and difference: Contesting the boundaries of the political (pp. 94-119). Princeton, NJ: Princeton University Press. Condorcet, J.-A.-N. d. C. (1785). Essai sur l’application de l’analyse à la probabilité des décisions rendus à la pluralité des voix. Paris: Imprimerie Royale. Csapo, E., & Slater, W. J. (1994). The context of ancient drama. Ann Arbor: University of Michigan Press. Davenport, T. H., & Prusak, L. (1998). Working knowledge: How organizations manage what they know. Boston, Mass: Harvard Business School Press. Dixon, N. M. (2000). Common knowledge: How companies thrive by sharing what they know. Boston: Harvard Business School Press. Estlund, D. M. (2003). Why Not Epistocracy? In Desire, Identity and Existence: Essays in honor of T.M. Penner (pp. 53-69). Academic Printing and Publishing. Estlund, D. M. (2008). Democratic Authority: A Philosophical Framework. Princeton: Princeton University Press. Fischer, F. (2009). Democracy and expertise: Reorienting policy inquiry. Oxford and


New York: Oxford University Press. Fishkin, J. S. (2009). When the people speak: Deliberative democracy and public consultation. Oxford: Oxford University Press. Furstein, M. (2008). Epistemic Democracy and the Social Character of Knowledge. Episteme: Journal of Social Epistemology, 5(1), 74-93. Garvin, D. A. (2000). Learning in action: A guide to putting the learning organization to work. Boston, Mass.: Harvard Business School Press. Gottlieb, P. (2009). The Virtue of Aristotle’s Ethics. Cambridge: Cambridge University Press. Hansen, M. H. (1987). The Athenian Assembly in the age of Demosthenes. Oxford: B. Blackwell. Keyt, D. (1991). Aristotle’s Theory of Distributive Justice. In D. Keyt & F. D. Miller (Eds.), A Companion to Aristotle’s Politics (pp. 238-278). Oxford: Blackwell. Kraut, R. (2002). Aristotle: Political philosophy. Oxford and New York: Oxford University Press. Landemore, H., & Elster, J. (Eds.). (2012). Collective Wisdom: Principes and Mechanisms. Cambridge: Cambridge University Press. Levitt, B., & March, J. (1988). Organizational Learning. Annual Review of Sociology, 14, 319-340. Lipset, S. M. (1959). Some social requisites of democracy: Economic development and political legitimacy. American Political Science Review, 53, 69-105. List, C., & Goodin, R. E. (2001). Epistemic Democracy: Generalizing the Condorcet Jury Theorem. Journal of Political Philosophy, 9(3), 277-306. List, C., & Pettit, P. (2004). An Epistemic Free-Riding Problem? In P. Catton & G. Macdonald (Eds.), Karl Popper: Critical Appraisals (pp. 128-158). London: Routledge. List, C. (2005). Group knowledge and group rationality: a judgment aggregation perspective. Episteme: Journal of Social Epistemology, 2(1), 25-38. List, C., & Pettit, P. (2011). Group agency: The possibility, design, and status of corporate agents. Oxford ; New York: Oxford University Press. Mackie, G. (2003). Democracy defended. Cambridge and New York: Cambridge University Press.


Marshall, C. W., & van Willigenburg, S. (2005). Judging Athenian dramatic competitions. Journal of Hellenic Studies, 124, 90-107. Mendelberg, T. (2002). The Deliberative Citizen: Theory and Evidence. Political Decision Making, Deliberation and Participation, 6, 151-193. Missiou, A. (2011). Literacy and democracy in fifth-century Athens. Cambridge: Cambridge University Press. Mutz, D., & Martin, P. (2001). Facilitating Communication Across Lines of Political Difference. American Political Science Review, 95(1), 97-114. Newman, W. L. (1887). The Politics of Aristotle [Commentary]. Oxford,: Clarendon press. Ober, J. (1998). Political Dissent in Democratic Athens: Intellectual Critics of Popular Rule. Princeton, N.J.: Princeton University Press. Ober, J. (2007). Natural Capacities and Democracy as a Good-in-Itself. Philosophical Studies, 132, 59-73. Ober, J. (2008). Democracy and Knowledge: Learning and Innovation in Classical Athens. Princeton: Princeton University Press. Page, S. E. (2007). The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton: Princeton University Press. Przeworski, A. (2000). Democracy and development: Political institutions and material well-being in the world, 1950-1990. Cambridge: Cambridge University Press. Reiter, D., & Stam, A. C. (2002). Democracies at war. Princeton, N.J.: Princeton University Press. Rhodes, P. J. (1985). The Athenian Boule. Oxford: Clarendon Press. Riker, W. H. (1982). Liberalism against populism: A confrontation between the theory of democracy and the theory of social choice. San Francisco: W.H. Freeman. Schmitt, C. (2004). Legality and legitimacy (J. Seitzer, Trans.). Durham [N.C.]: Duke University Press. Schultz, K. A., & Weingast, B. R. (2003). The Democratic Advantage: The Institutional Foundations of Financial Power in International Competition. International Organization, 57, 3-42.


Schumpeter, J. A. (1947). Capitalism, socialism, and democracy (2nd ed.). New York: Harper. Schwartzberg, M. (2010). Shouts, Murmurs and Votes: Acclamation and Aggregation in Ancient Greece. Journal of Political Philosophy, 18(4), 448-468. Sunstein, C. R. (2000). Deliberative Trouble? Why Groups Go to Extremes. Yale Law Journal, 110, 71-119. Sunstein, C. R. (2002). The Law of Group Polarization. Journal of Political Philosophy, 10(2), 175-195. Sunstein, C. R. (2007). Deliberating Groups versus Prediction Markets (or Hayek’s Challenge to Habermas). Episteme: Journal of Social Epistemology, 3(3), 192-213. Waldron, J. (1995). The Wisdom of the Multitude: Some Reflections on Book III Chapter 11 of the Politics. Political Theory, 23, 563-584. Wallace, R. W. (1997). Poet, Public, and ‘theatrocracy’: Audience Performance in Classical Athens. In L. Edmunds & R. W. Wallace (Eds.), Poet, public, and performance in Ancient Greece (pp. 97-111). Baltimore, Md.: Johns Hopkins University Press. Wilson, P. (2000). The Athenian institution of the Khoregia. The Chorus, the city, and the stage. Cambridge: Cambridge University Press.


TABLES 1. DEL 2. IGA 3. WEA Issue choice Deliberation, rules, or

exogenous Exogenous Deliberation/vote, rules, or

exogenous Options choice Rules or deliberation Exogenous Rules or deliberation/vote Independence No Yes No Updating Yes No Yes Judgment criteria Reasons offered by

deliberators Individual issue- assessments

Expertise or reputations & reasons by domain-experts

Decision by Consensus or vote Independent votes Domain-weighted votes Table 1 Three approaches to epistemic-democratic decision-making: Deliberation (DEL), independent guess aggregation (IGA), and weighted expertise aggregation (WEA). Rules = rules governing issues or options that are made democratically by the decision-making group. Exogenous = issues and options are given to the group by an authority external to the group, or by an existential threat to continued group existence. 1. Complete rules

infallible experts 2. Complete rules mass voting

3. Incomplete rules mass voting

Issue Rule Rule Exogenous or deliberation/vote Options Rule Rule Deliberation/vote Domains Rule Rule Deliberation/vote for closure Weighting Rule Rule Deliberation/vote for closure Experts # 1 per domain Multiple per domain Multiple per domain Voting by Domain-experts Mass on each domain Mass on options Judgment criteria

Individual expertise Reputation of experts, reasons given by experts

Reputation of experts, reasons given by experts

Decision aggregates

Domain-weighted ranking by experts

Votes of mass on each rule-weighted domain

Votes of mass on options

Table 2. Three versions of Weighted Expertise Aggregation

Parts Pw T1u T1w T2u T2w T3u T3w

Plot 8 3 24 1 8 2 16 Character 6 1 6 2 12 3 18 Thought 5 2 10 3 15 1 5 Diction 4 2 8 3 12 1 4 Song 3 3 9 1 3 2 6 Spectacle 2 1 2 3 6 2 4 Aggregate 12 59 13 56 11 53

Table 3: Hypothetical Aristotelian ranking of three groups of tragedies (T1, T2, T3). Pw = weighted value of a given part. Pw is multiplied by the unweighted rank (3=highest) to achieve the weighted value (w) of a performance. T(n)u = The unweighted rank of that part in a given tragedy group. T(n)w= the weighted value of a given part. Aggregate T(n)w (= the sum of weighted values) determines the correct (Aristotelian) ranking: T1>T2>T3.


VOTER Recognize parts

Identify experts

Assess experts

Weigh parts

Calculate aggregate Average

A 0.85 0.45 0.3 0.55 0.6 0.542

B 0.8 0.85 0.4 0.45 0.65 0.617

C 0.75 0.8 0.85 0.4 0.45 0.642

D 0.7 0.75 0.35 0.4 0.4 0.500

E 0.45 0.4 0.75 0.8 0.85 0.642

F 0.6 0.35 0.7 0.35 0.8 0.546

G 0.55 0.6 0.65 0.7 0.75 0.646

H 0.5 0.35 0.6 0.65 0.45 0.504

I 0.45 0.5 0.55 0.6 0.65 0.546

J 0.4 0.45 0.5 0.55 0.6 0.500

K 0.35 0.4 0.45 0.5 0.55 0.454

L 0.3 0.35 0.4 0.45 0.5 0.404

M 0.25 0.3 0.35 0.5 0.45 0.375

N 0.3 0.25 0.5 0.55 0.4 0.408

O 0.35 0.5 0.25 0.6 0.5 0.450

P 0.4 0.55 0.3 0.3 0.45 0.433

Q 0.65 0.4 0.75 0.4 0.25 0.496

R 0.5 0.55 0.4 0.35 0.3 0.425

S 0.55 0.5 0.45 0.55 0.35 0.488

T 0.6 0.55 0.5 0.45 0.4 0.504

U 0.65 0.45 0.55 0.5 0.45 0.521

V 0.45 0.65 0.6 0.55 0.5 0.554

W 0.45 0.4 0.65 0.6 0.55 0.538

X 0.45 0.75 0.3 0.65 0.6 0.563

Y 0.85 0.8 0.75 0.35 0.45 0.663

Average 0.53 0.52 0.51 0.51 0.52 0.52

Below 0.5 12 12 11 10 12 9

Table 4. Hypothetical distribution of citizen capabilities in five primary areas left unspecified due to incompleteness of rules. 0-‐1 scale of competence (likeliness to judge correctly) in each category.


NOTES. 1 Preferences and interests remain relevant in epistemic democracy, which assumes a shared interest in public goods, including (at least) security and welfare, and a preference for democracy over other regime types. These interests and preferences are treated as endogenous and assumed to be a matter of common knowledge. This set of assumptions may be most realistic when the stakes are high (existential threat) rather than low (division of spoils); see Ober 2008: 80-84. More generally, a realistic theory of epistemic democracy must be incentive compatible: ibid.: 5-22. Important recent work on epistemic democracy includes Cohen 1986, 1996; List and Goodin 2001; List 2005; Anderson 2006; Page 2007; Estlund 2008; Fursetin 2008; Fischer 2009; Schwartzberg 2010; List and Pettit 2011 (chapter 4); Elster and Landemore, 2012 (forthcoming). 2 Impossibility of collective rationality through preference aggregation under specified conditions: Arrow 1963. Riker (1982) argued that Arrow’s impossibility result, and the endemic possibility of cycling, rendered participatory democracy inherently incoherent; cf. Schumpeter 1947: 253, 269. Mackie (2003) argued against Riker that voting cycles seldom (perhaps never) occur in real world democracies. List and Pettit (2011) develop an Arrow-like impossibility result for judgment and attitude aggregation, but show how rational group agency is nevertheless achievable, employing non-mysterious reasoning grounded in methodological individualism. There are obviously some categories of decision (e.g. the truth value of mathematical propositions) for which no democratic process is plausible, regardless of its epistemic character. 3 Other potential problems with DEL include polarization, and cascading: Sunstein 2000, 2002, 2007; Mendelberg 2002; Mutz 2006. Paradox of majoritarian attitude aggregation: List and Pettit 2011: 43-47. 4 For discussion of this passage, see Newman 1887 III ad loc. (with citation of relevant comparanda from Aristotle and other ancient writers); Keyt 1991; Waldron 1995; Aristotle, Robinson, and Keyt 1995; Ober 1998: 319-26; Kraut 2002: 402-409; Gottlieb 2009: 200-207. 5 This section is reprised in similar language at 3.1286a24-31: Collective judgments are likely to be superior, “just as a feast to which many contribute is finer than a single and simple one, and on this account a mob (ochlos) judges many matters better than any single person.” 6 Waldron (1995) and Kraut (2002: 402-409) emphasize the passage’s deliberative character. Gottlieb (2009: 200-207) argues persuasively that Aristotle is neither being ironic nor presenting someone else’s argument here. She shows that the optimistic account of democratic decision-making in this and related passages is compatible with the discussion of the unity of the virtues in Nicomachean Ethics and Eudemian Ethics, because vices are disunited whereas the virtues (like collective judgments) cohere.


7 The centrality for Aristotle of the priority of the whole to its parts, and the political implications of that hierarchy: Politics 1.1253a20, 1260b14-15 with Ober 1998: 295-96. 8 Rather than, for example, a utilitarian outcome in which a bad experience of a few is overmatched by the good experience of many, or the mildly bad experience of many is overmatched by the superbly good experience of a few. Aristotle on justice as the common good: Politics 3.1278b20-25 with Gottlieb 2009: 201-202 n. 17. 9 The audience’s “vote” was in the form of what Schwartzberg (2010) calls acclamation. At the two major festivals, the tragic prize was for a set of three tragedies; the comic prize for a single comedy. Athenian audiences as judges of drama: Csapo and Slater 1994; Wallace 1997; Marshall and van Willigenburg 2005. 10 Cf. the passage preceding that cited in the text: “And since tragedy represents action and is acted by living persons, who must of necessity have certain qualities of character (êthê) and thought (dianoia)—for it is these which determine the quality of an action; indeed thought and character are the natural causes of any action and it is in virtue of these that all men succeed or fail— it follows then that it is the plot which represents the action. By ‘plot’ I mean here the arrangement of the incidents: ‘character’ is that which determines the quality of the agents, and ‘thought’ appears wherever in the dialogue they put forward an argument or deliver an opinion.” Poetics 1450a 11 In the terminology of List and Pettit (2011: 56-57) this is a strong (dictatorial) version of “distributed premise-based procedure,” which (per above) achieves rational coherence by relaxing both anonymity and systematicity. For a non-dictatorial version of distributed premise-based procedure, see below, n. 17. 12 Ranking based on unweighted parts. Plot: 1 3 2. Character: 1 2 3. Thought: 2 3 1. Diction: 2 1 3. Song: 1 3 2. Spectacle: 2 3 1. Aggregate : 2 1 3. Only the Character judge achieves the correct order. 13 Ranking based on aggregating weighted pairs of parts. Plot and Character: 3 1 2. Thought and Diction: 2 1 3. Song and Spectacle: 1 3 2 14 Ranking based on aggregating weighted groups of 3 elements. Plot and Character and Thought: 1 3 2. Diction and Song and Spectacle: 2 1 3. 15 I owe the opera critic and attending physician examples to discussions with David Large and Adrienne Mayor, respectively. 16 The mass of voters, in essence, does just what the opera critic or attending physician does: makes a judgment on each part, based on input from experts, and decides accordingly. In the terminology of List and Pettit (2011: 55-58) the procedure is “premise-based” (voting is on parts); it avoids irrationality by relaxing systematicity, while preserving weak anonymity.


17 It may seem unrealistic to expect each councilor to have a sense of the capabilities of 499 fellow councilors. But much of the actual deliberative work of the council was carried out by ten representative teams of 50 men (Ober 2008: 142-55), so it is quite plausible that each councilor had a more or less accurate “WEA capability table” of his team-mates in his head. The work of the 50-man teams, underpinning decisions of the full council, is a practical example of what List and Pettit (2011: 56-57) call “distributed premise-base procedure.” The comparison with a large academic department is again of relevance. 18 The gods’ will was revealed, cryptically, in responses to Athenian queries to Apollo’s oracle at Delphi regarding policy options. Gods, and the value of oracles, were taken by most Greeks to be facts about the world. These were not (we would say) brute facts about nature, but they were salient social facts that would have very considerable bearing on behavior. Distinction between social and brute facts, and high salience of social facts in the real world: Searle 1995. 19 In the terminology of List and Pettit (2011: 56-57) this is ultimately a “conclusion-based” procedure that preserves collective rationality by relaxing systematicity. Note, however, the essential work of the council, that precedes the decision in the assembly, makes ample use of distributed premise-based procedure. 20 The Athenian citizenry had decided to fight at Marathon ten years before, but the situation was quite different: the response was to a much smaller sea-borne attack, and so the familiar option of sending out the infantry in full force to repel invaders was a reasonable (and as it turned out successful) response. The Athenians emphatically did not agree with Schmitt (2004 [1932]) that exceptional circumstances, in which the state itself is at risk, demand the suspension of democratic deliberation and voting in favor of a dictator. 21 Democracy and economic performance: Lipset 1959; Przeworkski 2000; Boix 2003; Acemoglou and Robinson 2006. Performance in the inter-state arena: Reiter and Stamm 2002; Schultz and Weingast 2003. 22 Another reason may be the inherent difficult of weighing the benefits of good choices against the costs of bad ones.

weighted expertise aggregation: an aristotelian middle … · weighted expertise aggregation: an...

Documents