engaging experts: dealing with divergent elicited priors...
TRANSCRIPT
Engaging Experts:Dealing with Divergent Elicited Priors in Political Science
Sarah B. Bouchat
15 October 2016
Not For Circulation or Citation
1 Introduction & Motivation
Priors over parameters constitute one of the most visible manifestations of the theoretical distinc-
tions between Bayesian and frequentist statistics. The information contained within the specified
prior, however, will depend upon the knowledge available to a given researcher. An elicited
prior seeks to leverage the substantive knowledge of area experts, whether through interviews
or published research, in order to improve the accuracy of posterior estimates. Elicited pri-
ors as a concept have been detailed since the early work of Savage (1971), Shuford and Brown
(1975), Kadane and Wolfson (1998), and Garthwaite, Kadane, and O’Hagan (2005). Formally,
Gill and Walker (2005, 841) discusses the elicited prior “as a means of drawing information from
subject-area experts with the goal of constructing a probability structure that reflects their specific
qualitative knowledge, and perhaps experiential intuition, about the studied effects.” These aims
and applications for elicited priors seem both reasonable and desirable, yet the 10 years since
the publication of Gill and Walker’s article have seen few instances of researchers, especially
in political science, adopting elicited priors as a way to incorporate qualitative knowledge into
quantitative research.
This paper argues that the limited application of elicited priors is due to both an unclear
prescription for implementing the method and a limited vision for its application. The following
sections seek to explore the range of studies from other disciplines that use elicited priors in
order to understand both how they are used in practice and how they might be better used
to realize the vision of qualitative and quantitative synergy that Gill and Walker present. An
empirical section also outlines a specific process for undertaking work using elicited priors. This
is illustrated with reference to applications that expand Gill and Walker’s scope to encompass
data-poor and authoritarian contexts in particular, where the limitations on quantitative data
1
especially benefit from an elicited priors approach.
The concept of elicited priors shows particular promise in combining qualitative and quan-
titative analytical approaches. A typical modeling approach would rely only on the researcher’s
assessment of the appropriate array of variables, model type and structure, and method. Elicit-
ing priors from those with expertise in an area offers an opportunity to gather more input for
each of these choices from those who have substantive knowledge but are not directly related
in the process of analysis. This approach has the added benefit, therefore, of maintaining trans-
parency about assumptions in the modeling process. An elicited prior requires documentation of
at least the qualifications of the source, whether in-person or published, and can therefore ease
the process of replication and progression in similar veins of research.
Despite these advantages, the current work on elicited priors lacks guidance on how to
elicit said priors, how to reconcile diverging beliefs, and under what circumstances or for what
purposes they would be most useful. These omissions limit the use of elicited priors in the social
sciences, and especially in political science. In particular, while the prospect of eliciting priors
from experts with both experiential and scholarly insights adds to the appeal of the method, it is
exactly the process of identifying “experts,” eliciting their views in ways that accurately represent
their assessments while being useful in the modeling process, and reconciling the diverging opin-
ions of experts that makes elicited priors difficult to apply. This is especially true in authoritarian
and poor data contexts, where the prior is simultaneously of greater importance to the mod-
eling process and more contentious to construct as a result of the data-poor environment. For
example, when soliciting input from government-sponsored and opposition-sponsored experts
in authoritarian settings, there are no guidelines inherent to using elicited priors as presented by
Gill et al. that would allow a researcher to adjudicate between strongly diverging opinions or
assess their credibility. Likewise, no guidance is given regarding precisely how to elicit a prior—a
process that could greatly influence outcomes.
This paper seeks to address these omissions and suggest ways in which elicited priors
could be usefully applied, particularly in studies in authoritarian or data-poor contexts. The
following section culls techniques from the literature using elicited priors to create suggestions
for their broader application in social science. An empirical section follows, discussing strategies
2
for implementing elicited priors and their implications in a situation with simulated data. Finally,
the conclusions section will offer some next steps for refining these approaches and for broader
application of elicited priors to a wider array of social scientific problems.
1.1 Priors in Bayesian Analysis
Often, researchers seeking to remain agnostic in the statistical analysis employ relatively diffuse
priors. These “uninformative” priors are, however, literally informative at least two senses. First,
they directly influence the conclusions to be drawn from the modeling process. This is especially
true in cases of low data quality or quantity: diffuse priors limit the ability of the researcher
to leverage the model structure in order to draw inferences about empirical phenomena. As
Andrew Gelman writes, “[with] well-identified parameters and large sample sizes, reasonable
choices of prior distributions will have minor effects on posterior inferences. ... If the sample size
is small, or available data provide only indirect information about the parameters of interest, the
prior distribution becomes more important” (Gelman 2002, 1634). While other approaches (e.g.,
clustering and estimating hierarchical models) often seek to address these issues, these state-
ments still justify overarching concern with specifying reasonable priors where possible, with
special consideration for instances with poor data. Specifying noninformative priors, further-
more, threatens the trajectory of scientific inquiry as studies build on each other, as Gill and
Witko note: “It is also important to observe that the overwhelming proportion of prior distribu-
tions specified in published Bayesian social science work still avoids using reasonably informed
priors, which unfortunately hurts the steady accumulation and progression of scientific knowl-
edge” (Gill and Witko 2013, 462).
Second, these priors represent a strong positive claim that no useable information about a
given θ exists with which to specify a more appropriate or precise prior. More generally, this ap-
proach reflects a normative position that favors the omission rather than the careful specification
of assumptions throughout the modeling process.
An alternative approach would instead seek to take advantage of the significant knowledge
accumulated across fields of study in order to select appropriate, informed priors. From the per-
spective of the researcher, this approach could represent an alternative agnosticism that does not
require reliance on own knowledge or opinions, but rather documents the acquisition of infor-
3
mation from expert sources. This notion effectively underlies the method of “eliciting” priors:
whether through interviews with area experts or reference to published works, a researcher can
develop appropriate priors for unknown θ that improve the prospects for modeling in a Bayesian
framework.
2 Literature Review: Elicited Priors across Disciplines
The notion of elicited priors is not new, yet scholarship examining or employing the technique
remains limited. Searching for “elicited prior” in the Web of Science database yields fewer
than 50 relevant articles, most of which have substantive foci in biology or psychology rather
than political science. In defining the origins of elicited priors, Gill and Walker (2005) reference
previous terms such as “community of priors”—which incorporate opinions of both affirming
and skeptical experts—and delineate elicited priors into a variety of types, including clinical
priors, skeptical and enthusiastic priors, reference priors, etc. (843). Gill and Walker (2005, 844)
summarize the three phases of research using elicited priors explained by Spetzler and Stael von
Holstein (1975) as the deterministic, probabilistic, and informational phases. The deterministic
phase, encompassing variable and expert selection, has costs, but is generally perceived as less
challenging than the probabilistic phase, during which priors are actually elicited. Aside from
determining data sources, the primary question addressed in the first stage is instead from how
many experts one should elicit priors (Gill and Walker 2005, 844).
The informational phase follows, in which elicited responses are tested, evaluated, and
scaled for consistency. Gill and Walker (2005) concur with Spetzler and Stael von Holstein (1975)
that the methods of eliciting priors (“p-methods,” “v-methods,” “pv-methods,” etc.) present the
greatest challenge, but concerns about how best to elicit priors from experts do not explain the
very limited uptake of elicited prior methods overall in the social sciences. Rather, I suggest
that more careful attention to the selection of “experts” and the calibration of responses opens
possibilities for applying this method to new areas of research, specifically in authoritarian and
data-poor settings. For example, rather than focusing on the purely statistical problem of how
many experts should be selected, emphasizing the question of who is considered an expert and
in what way opens new possibilities for identifying expertise and knowledge from which to
generate priors.
4
2.1 Theoretical Contributions to Elicited Priors
While the use of elicited priors in scientific fields far surpasses its use in social science domains,
its application remains somewhat limited by remaining skepticism of the “subjectivity” of the
Bayesian approach more generally (Lele and Das 2000; Wang and Zhou 2009). Elicited priors,
rather than overly diffuse or benchmark “objective” priors, have the potential to provide leverage
in low-data circumstances and challenging modeling contexts, but also to substantively impact
the quality of estimation in a positive way. In fact, Datta and Ghosh (1991) suggest that the
elicited prior represents the “true” prior—presumably that, if sufficient expertise were available
and applied, the resulting prior would be accurate. Still, problems remain with processes for en-
gaging even trained individuals in statistical assessments. As Lin, Lin, and Raghubir (2004) note,
people are susceptible to biases resulting from self-positivity, controllability of negative events,
and especially order of elicitation—where previous questions are used to form assessments for
later ones. Likewise, practically speaking, prior elicitation confronts the same issues as typical
Bayesian analysis, where, for example, specifying priors for continuous parameters proves dif-
ficult since “[doing] so would require infinitely many prior probability judgments” (Dey and
Birmiwal 1994).
In a sense, the lack of identifiable expertise in low-data settings, rather than giving rise
to the implementation of diffuse priors, should give rise to a concerted effort to model our
ignorance—a task Zaffalon (2005) identifies as a persistent challenge in Bayesian analysis. Ac-
counting for prior ignorance and incomplete data, as Zaffalon explains, means that the true
model of ignorance requires accounting for “all possible states of knowledge” (1005). One of
the key related areas of inquiry in elicited priors is how to handle uncertainty or error within
the prior itself. In general, studies posit an elicited prior π0 in which there may be some “ε
contamination” or error (Sivaganesan 1993).
Among these scientific papers, however, opinions about how best to elicit priors vary. Lele
and Das (2000), for example, argue in favor of directly soliciting guess values:
It is important to recognize that the concept of a prior probability distribution onthe parameters of a statistical model is a statistical construct that is hard for mostscientists to visualize. It is more natural for an expert to think in terms of the process
5
under study and not in terms of statistical distributions over a parameter space. Thesensible approach, then is to ask the expert to provide guess values for observabledata, not a prior probability distribution. (Lele and Das 2000, 466)
In Lele and Das (2000)’s case, this suggests a hierarchical modeling strategy to combine experts’
insights with the observed data under study.
The problem confronted by Lele and Das (2000)—sparse data in a spatial context but the
potential for “soft” data in the form of expert opinion—mirrors the difficulty of conducting analy-
ses in authoritarian or low-data situations. Lele and Das (2000) provide an analytical framework
for addressing precisely this problem of “misleading” expert priors in their paper examining
spatial hierarchical models with elicited priors. In particular, the authors want to account for
the dependence between observed data values and elicited prior values when experts are asked
for value estimates rather than providing complete prior distributions (468). “This dependence
reflects the credibility of the expert,” they argue, while noting that “[in] this way, even a mislead-
ing expert opinion can be informative” because “if we find that data elicited from an expert are
negatively correlated with real data, this is useful information that can be used to suitably adjust
our inferences” (468). To do this, Lele and Das imagine that an expert gives an opinion E about
Y drawn from the distribution f(y; θ) where the θ parameters are unknown. This opinion E is
itself distributed g(e | y;η), with η indicating the dependence between Y and E. This leads Lele
and Das to call η an “honesty parameter,” measuring the “credibility” of an expert. In principle,
however, such a parameter captures both the ability of the expert to think and express opinions
statistically as well as the potential strategic misrepresentation of information and expertise.
2.2 Empirical Applications of Elicited Priors
A variety of disciplines, although primarily within the natural and biological sciences, have
utilized elicited priors techniques for research. These studies offer a multitude of insights about
how to implement an elicited priors approach, but also highlight the ways in which current
practice in other disciplines is not currently well-suited to the study of authoritarian or low-data
contexts.
Morris et al. (2013), for example, investigate the effects of vesicoamniotic shunts on lower
urinary tract issues for fetuses where they have only 31 female subjects with singleton pregnan-
6
cies and leverage elicited priors from 52 pediatric nephrologists, pediatric urologists, and fetal
medicine specialists (Morris et al. 2013, 1500). The experts polled in the study largely agreed on
the potential effects of treatment, although the authors note that it is “problematic” that these
experts’ opinions did not align with what is current common clinical practice. Their responses
were pooled and averaged to create a prior distribution (1501). This study provides one of several
examples in which expert opinions are equally weighted in the implementation of the prior. For
social science purposes, however, two issues remain:
(1) How should expert opinions be weighted or aggregated if disagreement occurs?
(2) How should expertise coming from practical experience be reconciled, either with differing
experience or with educated or credentialed expertise?
While studies like Morris et al. (2013) illustrate the disjuncture between expert opinion and
practice, they do not resolve the concerns facing researchers who wish to use these methods in
other social scientific settings.
The procedure for eliciting priors from experts also dialogues with these concerns for how
expert priors might be adequately reconciled and combined. As Wheeler et al. (2014) describes,
there are many possible ways of “eliciting” prior information. This can be done with previously
published work or historical data as well as with interviews of living experts; the primary binding
constraint is that the expert has not directly observed the data under current study (678). If the
expert has directly observed the data in question, their prior is likely to reproduce these data
rather than providing additional information that can aid in inference and analysis. Despite this
constraint, there are still many options for how to implement prior elicitation, including eliciting
priors about regression coefficients; the distribution of the dependent variable conditioned on
fixed values of covariates; quantiles of the predicted dependent variable distribution, etc. (678).
Some of these methods present greater challenges than others in implementation. For
example, eliciting beliefs regarding regression coefficients and their variance can prove difficult
even for educated experts (Gill and Witko 2013, 462), as providing accurate estimates of statistical
uncertainty of one’s beliefs is a particular challenge (Albert et al. 2012, 504). A variety of methods
also exist for quantile based direct elicitation of priors (Dey and Liu 2007). Regardless, as Wheeler
et al. (2014) notes, “the incorporation of such information into the Bayesian modelling framework
7
aligns with the philosophy of the scientific method, where knowledge that is available before
collective data (prior) is used along with the observed current data (likelihood) to inform what
we know now (posterior)” (Wheeler et al. 2014, 678).
The example offered by Daponte, Kadane, and Wolfson (1997) underscores this point. The
authors use elicited priors to project the Iraqi Kurdish population from 1977–1990, and describe
three critical advantages to conducting their demographic study in a Bayesian framework. First
and foremost, they note that a Bayesian approach can promote communication among demo-
graphic researchers: “Making one’s beliefs explicit using probability distributions allows other
demographers to observe exactly how one views the sources of uncertainty in the phenomenon.
Others can then know on what they agree or disagree. The reasons given for particular prob-
ability distributions can be an important source of insight” (1256). The authors also note that
making these projects explicit in the form of probability distributions enhances their usability in
a variety of applications. Finally, they distinguish the Bayesian approach and its advantages from
more traditional methods: “Classical models either include or exclude a parameter about which
no prior is expressed, which is often equivalent to expressing certainty about its value. Using
probability distributions permits one to express states of knowledge in between these two alter-
natives” (1256). Much like other social scientific applications, as the authors describe, this study
of the Iraqi Kurds “lacks high-quality data” and reflects incomplete information—conditions that
align it in particular with the study of authoritarian regimes or other low-information environ-
ments in political science (1257). Each of these dimensions highlights the critical applicability of
an elicited priors approach in an authoritarian or low-data context, and the significance of elicited
priors as a source of information to adapt and improve statistical inference. How to integrate the
priors offered by experts as part of a study, however, remains unaddressed by this work.
Albert et al. (2012), on the other hand, tackle the issue of how best to aggregate elicited
priors directly, proposing a hierarchical modeling approach to account for multiple sources of
variation and the potential lack of independence between experts’ assessments (503). For illus-
trative purposes, the authors define a sampling model containing observation X ∼ Pθ where θ is
an unknown parameter with prior π that is an “elaboration on a parametric family” such that
π ∈ {πγ,γ ∈ Γ }. γ is then estimated from the elicited priors (504). It is plausibly the case, as
the authors note, that each expert polled provides a different γ, necessitating a procedure to
8
combine these divergent priors. In the authors’ review, pooling and averaging predominate as
methods for combining these differing priors, where averaging “emphasizes the consensus on
elicited quantities” and is advantageous in its “simplicity,” but at the same time can “understate
variation by ignoring uncertainty” and/or “mis-represent [sic] multiple modes” (Albert et al.
2012, 504). Pooling methods—whether linear or logarithmic—attempt to overcome some of these
deficiencies by encompassing all values in a way that can be construed as an additive or multi-
plicative mixture (504). These divergent priors can then be combined using weights w`, such as
by∑L`=1w`πγ` where ` indexes the individual expert whose prior πγ is being combined with
others (504). Devising these weights, however, as the authors report, has primarily been based
on “p-values for evaluating how well expert assessments on seed variables align with empiri-
cal results” (504). This has the disadvantage, as the authors note, of embracing the “diversity”
among the elicited priors without offering a notion of consensus or how individual experts might
diverge from that consensus. More directly related to the study of authoritarian regimes and ex-
perts, however, this method of weighting expert opinions is intended to resolve discrepancies
in experts’ ability to offer statistical assessments or measures of uncertainty. In a situation of
incomplete information both within the data and among the experts, assessing an individual
expert’s prior in alignment with the data may less appropriately capture their insights into the
underlying data-generating process or motivations of actors captured within the data. Weighting
expert opinions in this way, rather than understanding as well their potential incentives to offer
misleading information or ability to only offer insights into one part of the “truth,” means that
existing methods for combining expert opinions are not well suited to use in low-data contexts
or the study of authoritarianism in particular.
The method that Albert et al. (2012) propose essentially establishes a hyperprior on the
unknown parameter θ. This process treats elicited information as “data” in the construction of
a prior for the eventual analysis of interest. The prior probability distribution on θ, given by
Delicit gives rise to the following:
π(γ | Delicit) ∝ f(Delicit | γ)π0(γ)
The construction of the distribution of π(γ | Delicit) from pooling requires a joint likelihood
9
of expert opinions in order to account for dependence. The resulting model of the expert priors
treats prior opinions as being sorted into classes Jwhere members of the same class have the same
distribution (Albert et al. 2012, 508). In this the authors acknowledge that smaller groups may
represent the divergent opinion of a set of experts who are less represented in the population, for
example, or who are less reliable, and the solution they propose is to assume a higher variance
parameter for the distribution of a group j with those characteristics. This approach, while
including important information or knowledge about the experts into the modeling exercise,
relies on assumptions about the validity of the statements made by members of smaller groups of
experts, which may not reflect a priori knowledge of experts in authoritarian settings. Likewise,
this conception of groups of expert opinions is predicated on an implicit notion that some of these
elicitations are accurate whereas others are flawed, rather than a conception that each contains
some aspect of the true information.
O’Leary et al. (2009) reinforce that the elicitation method itself should reflect knowledge
about the experts who might provide information; that is, it should take into account difficulties
in engaging with experts as well as difficulties with experts providing accurate information.
“Selection of an elicitation method,” they write, “is determined by several issues. These include
the expert’s knowledge of statistics, their mapping skills, time available, access to experts and
funding. The chosen elicitation method should balance the expert’s knowledge of statistics and
mapping with the output required” (396). These considerations serve as the basis for three
different elicitation methods the article discusses, each with respect to a dataset concerning rock
wallaby population estimates. The choice of elicitation tool, the type of elicitation (e.g., indirect P-
method, direct, etc.), and the distribution form of the β coefficient primarily distinguish between
these methods (388). This distinction is useful in highlighting that many methodological choices
already exist in terms of adapting an elicted priors approach to a particular research question
and context, but none specifically address the problem of how to aggregate priors across experts
beyond the method of weighting with p-values for alignment between the elicited prior and the
data that were discussed previously.
These problems notwithstanding, a precedent does already exist within political science
for engaging “experts” beyond the credentialed class often sought after by scientists. Kendall,
Nannicini, and Trebbi (2015) elicit multivariate distributions from voters reflecting their beliefs
10
about incumbents’ valence and ideology. This engagement of “experts” from a more general
populous can allow the ultimate prior distribution used in the analysis of interest to reflect the
diversity of information about the subject. Likewise, Small (2008) acknowledges that “it may
be appropriate in many cases to elicit probabilities from different experts for different parts of
the model” (1302). Likewise, Bakker (2009) utilizes elicited priors to enable experts to refine
political party placements on a left–right spectrum. The broader application of an elicited priors
approach to important problems in political science, however, rides on the ability of the approach
to be adapted to other contexts and questions. The next section discusses how in particular this
approach should be beneficial for the study of authoritarian political processes, and where its
current practice falls short in offering clear signs of applicability.
3 Empirical Applications to Authoritarian and Low-Data Contexts
The aforementioned studies often engage expert opinion for priors in order to leverage the greater
precision this lends to the analysis. An analogous application in political science relates to ques-
tions concerning authoritarian regimes or developing country contexts where low data and data
manipulation/falsification are present. Bayesian analysis is often most appropriate for small-n
studies, and more informative priors from elicitation can aid in resolving pathological prob-
lems arising from small data or complex modeling structures. This area therefore seems like a
straightforward one in which to apply an elicited priors approach, but the current literature has
several shortcomings that prevent the direct importation of elicited priors into a social science
domain—especially where the study of authoritarian regimes or low-data contexts is concerned.
First, while some of the above studies offer more detailed accounts of how to elicit pri-
ors for scientific applications, these accounts have not been translated to a social science setting.
This would be particularly important when expanding the notion of “expertise.” Elicited priors
are commonly thought to come from “experts” in a given field. While this may be a sufficient
distinction in fields like medicine, where credentials come through schooling and certification,
it is less clear for social science applications. When trying to understand regime behavior un-
der authoritarianism, for example, one could conceivably ask professors who study the subject,
bureaucrats who work under those conditions, activists who oppose such a system, or average
citizens who live under authoritarian rule. Each of these groups has some degree of “expertise”
11
to offer, but each would likely also have varying levels of statistical literacy to align with the
protocols developed for eliciting priors in scientific contexts.
A second, and related, problem is the issue of diverging priors. The common procedure
in the scientific literature, although not often discussed in detail, is to average elicited priors or
to use a “consensus prior” after having elicited information from a series of experts. These ag-
gregation rules intentionally eliminate or deemphasize information that diverges from the most
common or popular prior. This may make sense for a panel of credentialed experts in a scientific
field: for example, a researcher may be interested in the “state of the field” where the effect of
a medication on prognosis is concerned. This rule makes less sense, however, in a circumstance
like the one described previously, where the notion of expertise has been expanded to encompass
several valid sources of information from a social scientific perspective. Particularly in authori-
tarian regimes, these different individuals may have widely varying perspectives on the issue at
hand. These differing perspectives could be a result of different experiences, different relation-
ships to the ruling regime or party (co-opted or oppositional), or different access to information
in what is likely to be an environment rife with incomplete information.
A comprehensive analysis seeking to leverage this expertise nonetheless needs more guid-
ance on how to incorporate and assess these divergent priors. Averaging the priors would dis-
card potentially significant information, and arriving at a “consensus” could prove theoretically
problematic. For example, imagine there are two individuals whose priors roughly overlap or
correspond, but a third whose prior is significantly distinct. Should the researcher treat the two
that correspond as the “consensus” or should the researcher be skeptical that those individuals
have reproduced some sort of “party line” that intentionally obscures the underlying statistical
relationship, whereas the third individual is telling the “truth”? A more ecumenical approach
would seek to incorporate each of these, with the understanding that each potentially contains
some part of an overall “truth.”
Particularly when discussing social and political phenomena, “experts” may have strongly
divergent opinions about the relationship between covariates and outcomes that, unlike in some
scientific fields, cannot be resolved immediately by appealing to the data. Some of these experts
may be more reliable than others in the sense of being able to provide statistical assessments,
12
but in authoritarian contexts in particular, situations may arise in which experts have differing
but not apparently false assessments of statistical likelihood as a function of either their desire
to selectively provide (or misrepresent) information or their differing perceptions of truth caused
by skewed information environments in authoritarian regimes.
In the following sections, I present an approach for addressing these concerns by providing
an aggregation tool that can synthesize divergent expert opinions for Bayesian analysis. This
approach does not seek to improve on methods for providing internal, statistical consistency of
elicited priors, but rather seeks to deal with the divergence in potential elicited priors and the
possible propensity for experts in authoritarian contexts to misrepresent the relationship between
covariates and outcomes.
3.1 A Dirichlet-Based Method for Elicited Priors
In this section, I propose an approach for resolving the dilemma of prior elicitation and aggre-
gation in social scientific contexts, and especially in studies of authoritarianism. Following this
discussion, I provide examples of other uses for Dirichlet approaches as well as an example of
an empirical circumstance where the approach could be applied with respect to elicited priors.
The method proposed in this paper adapts a Dirichlet process in order to resolve the issue
of aggregating divergent elicited priors before implementing a Bayesian analysis. This Dirichlet-
based method allows for multiple possible aggregation rules for elicited priors, enabling each of
the priors to be “mixed” in a single distribution, and varying the concentration parameter of the
Dirichlet distribution to seek a final prior that emphasizes greater consensus, equally weights
all opinions, averages over opinions, etc. As Neal (2000) describes, “[mixtures] with a countably
infinite number of components can reasonably be handled in a Bayesian framework by employing
a prior distribution for mixing proportions, such as a Dirichlet process, that leads to a few of
these components dominating” (249). This flexible framework for establishing prior distributions
that carry substantial significance in a low data context is aimed to improve the integration
of qualitative knowledge into quantitative assessments of particularly challenging contexts and
questions. This approach facilitates a greater number of “new” kinds of experts serving as
subjects for prior elicitation. Incorporating these differing kinds of expertise both enables better
use of the information and expertise that already exists in authoritarian and other low-data
13
contexts, as well as offers an opportunity for better engagement with qualitative researchers,
who have long utilized this type of expertise in their own work.
The Dirichlet distribution is defined as
1
B(α)∏Ki=1 x
αi−1i
where α specifies a concentration parameter for the K “clusters” of the distribution. The Dirichlet
as a mixture model of mixtures can be thought to play a “partitioning” role for clusters of obser-
vations. In the Chinese restaurant process example illustrating a Dirichlet, a customer entering
the restaurant can be seated at a table already occupied by some diners, or at a new table. The
probability of being seated at a given table corresponds to the distribution of occupants at one
table relative to the others.
Applying this process to the elicited priors case, each elicited prior is separated into its
constituent components: the proposed value for the dependent variable y as well as the values of
independent variables x that corresponded to the y value. These proto “distributions” are treated
as data for the purposes of the Dirichlet process. The Dirichlet will dynamically create clusters
of observations, essentially finding moments of consensus since the population of these clusters
is weighted by the existence of other data to occupy the cluster. Through an updating process
to create these clusters, the Dirichlet process will eventually provide posteriors corresponding to
each expert from whom a prior was elicited. These posteriors can then be used to create a prior
distribution for the eventual analysis of the data of interest.
This framework is flexible, however, in terms of the specification and in terms of how input
is weighted. An explicitly symmetric Dirichlet, which can be expressed as:
Γ(αK)
Γ(α)K
K∏i=1
xα−1i
would denote that all components of α are of equal value (equal weighting). Likewise, how the
posterior information from the Dirichlet process is transformed for use as a prior in the data
analysis leaves room for defining different aggregation rules: averaging, deemphasizing extreme
values, deemphasizing consensus, etc. This process of influencing the aggregation itself reflects
14
an implicit hyperprior by the researcher placed on the priors, and the aggregated prior, from the
experts used for elicitation, specifying how much weight to place on different experts’ opinions
or on different components of the aggregation.
This proposed method of aggregation adds significant value relative to the existing meth-
ods of pooling or averaging. In addition to being better equipped to deal with the potentially
divergent priors elicited in political contexts relative to medical or biological fields of study, this
method provides a formal and transparent way to incorporate the researchers’ beliefs about the
credibility of a source, and to weigh the credibility of sources against each other and incorporate
them into the final prior. This method also has the distinct potential to engage scholars across
the qualitative-quantitative divide, as it does not resolve issues of expert selection but instead
requires relying on the experience and knowledge of qualitative scholars who have previous
familiarity with the case at hand to identify “expert” individuals and to evaluate their credibility.
3.2 Dirichlet Process Example
A common application of Dirichlet processes is in Latent Dirichlet Allocation (LDA). This process
provides a model of a corpus, or body of textual works, where each document comprises a
random mixture of latent “topics” and topics are distributions of words (Blei, Ng, and Jordan
2003, 996). For the basic LDA model with documents w in corpus D, as described by Blei, Ng,
and Jordan, you select N ∼ Poisson(ξ), θ ∼ Dirichlet(α), and for each N words in wn, you
choose a topic zn ∼ Multinomial(θ) and a word wn from p(wn | zn,β) (996). In this case the
mix over latent topics θ is given with a Dirichlet having a concentration parameter of α. The
ultimate allocation process seeks to understand how documents, which vary according to their
words, can be sorted into a variety of topics. This is analogous to the mixture and allocation
process that would occur in the elicited priors application of the Dirichlet process, where latent
states correspond to experts’ hidden dimensions of consensus. That is, the elicited priors are
analogous to documents in the LDA example, which are comprised of components used to elicit
them (e.g., across coefficients, according to quantiles, etc.). Experts offering these priors may have
been subject to the same kinds of constraints (e.g., career or family concerns) or had access to
or limitations from the same kinds of information (e.g., censorship, governing versus governed
classes) that would shape the prior they gave. Allocating these into latent classes using the
15
Dirichlet allows the researcher to examine the groupings present in the elicitations in order to
better weight and incorporate the contributions of each individual expert.
4 Dirichlet Allocation Example
One of the key contributions of this method is to allow researchers seeking to utilize expertise
from a diverse and perhaps divided set of individuals to be able to aggregate and incorporate
that divergent expertise into their prior distributions. The Dirichlet-based approach allows for
substantively similar elicited priors to be clustered for more efficient estimation and a more
representative final prior. Adequately capturing this diversity within the Dirichlet, however,
depends on several parameters that ultimately come from the initial research design, including
how many experts to engage and how many questions to ask and/or datapoints to collect for
each covariate used in the final analysis.
In order to be able to calibrate the process of selecting a number of experts and how to elicit
priors while also optimizing the construction of the aggregated prior, the Dirichlet allocation
was moved to C++. The figures that follow illustrate how well the Dirichlet approach identifies
clusters among priors as a function of the number of covariates and the number of questions
asked. As these figures illustrate, the clustering process performs best when receiving enough,
but not too much, information about experts’ priors. For example, it performs equally well in
returning the true clustering with 20 experts versus 10, as long as there were 10 questions for
10 covariates. Slightly greater uncertainty is introduced, however, when 15 questions are asked
for 4 covariates, and even more uncertainty appears when fewer (10) questions are asked for 4
covariates. This uncertainty appears in the figures as gradations of light gray or light turquoise,
whereas when two experts share a cluster their cell is turquoise, and when they do not, their cell
is dark gray.
For example, in Figure 1, 20 experts are allocated into 5 clusters, where for example cell (1,1)
is shaded turquoise because expert 1 shares a cluster with herself. Figures 2 and 3 demonstrate
more uncertainty than Figure 1. In Figure 2, much more information is gleaned from the experts
than corresponds to the lower number of covariates, leading to too fine-grained information for
sorting. Figure 3’s configuration performs better with fewer questions per expert.
16
5 10 15 20
51
01
52
0Estimated Probability of Same Cluster
Expert
Exp
ert
5 10 15 205
10
15
20
Truth of Whether in Same Cluster
Expert
Exp
ert
Figure 1: Cluster Allocation with 20 experts, 10 questions, 10 covariates
2 4 6 8 10
24
68
10
Estimated Probability of Same Cluster
Expert
Exp
ert
2 4 6 8 10
24
68
10
Truth of Whether in Same Cluster
Expert
Exp
ert
Figure 2: Cluster Allocation with 10 experts, 15 questions, 4 covariates
17
2 4 6 8 10
24
68
10
Estimated Probability of Same Cluster
Expert
Exp
ert
2 4 6 8 10
24
68
10
Truth of Whether in Same Cluster
Expert
Exp
ert
Figure 3: Cluster Allocation with 10 experts, 10 questions, 4 covariates
18
5 Dirichlet Process Aggregation Applied: Western and Jackman (1994)
In order to illustrate the flexibility of this approach in incorporating multiple divergent priors,
I take up the analysis conducted by Bruce Western and Simon Jackman in their 1994 paper,
“Bayesian Inference for Comparative Research.” In the paper, the authors highlight the many
benefits of Bayesian analysis for comparative politics research. In particular, they emphasize that
the use of priors to encapsulate differing ideas about the state of the world and combine them
with data to draw inferences formalizes a process already undertaken in comparative politics
research, albeit less transparently and with less ability to directly adjudicate between competing
views. The authors illustrate this argument by drawing on a then-recent debate between Michael
Wallerstein and John Stephens concerning the most important factors giving rise to unionization
in advanced industrialized democracies.
Western and Jackman’s aim is very similar to my project in this paper: namely, to empha-
size the importance of priors as instantiations of knowledge, and to highlight the challenge of
dealing with differing perspectives even within a prior probability framework. In the context of
the Wallerstein/Stephens debate, this concern is applied to the study of union density—union
members as a percent of the labor force. While Wallerstein argued for the size of the labor force
as the most critical determinant, Stephens favored industrial concentration as an explanation. As
Western and Jackman note, these two variables are nearly perfectly negatively correlated (Jack-
man and Western 1994, 416). Furthermore, the sample of interest—20 industrialized nations in
a single year—both suggests against a frequentist approach and increases the challenge of the
collinearity between the favored explanatory variables. Drawing on this debate, and with ref-
erence to outside sources, Western and Jackman construct plausible prior means and variances
for Stephens and Wallerstein for each of three explanatory variables relative to unionization: left
government, log labor force size, and economic concentration. Each of these prior means and the
corresponding precisions are represented in the figure below.
The most notable difference, as suggested by their written works, is that Wallerstein be-
lieves logged labor force should have a strong negative effect on unionization (Stephens believes
it has no effect) and that in turn Stephens believes economic concentration has a strong positive
effect on unionization, whereas Wallerstein believes it has none.
19
Intercept Left Govt Log Labor Econ Concentration
Stephens
Wallerstein
-0.50 -0.25 0.00 0.25 0.50-20 -10 0 10 20 -5 -4 -3 -2 -1 0 0.0 2.5 5.0 7.5 10.0Prior Mean and Precision
Expert
Figure 4: Wallerstein and Stephens’ Priors and Precisions as Shown in Jackman and Western(1994)
In the subsequent analysis, Jackman and Western use each of these priors respectively to
conduct a simple Bayesian linear regression
Unionization ∼ α+ Leftgovt+ LogLabor+ EconConcent+ ε
using 20 observations of country-level data on unionization and each of the explanatory variables.
They first estimate the regression with uninformative priors to find a baseline result, and then
apply Wallerstein’s prior and Stephens’ prior separately in turn to compare the results. Their
findings illustrate both the power of priors to shape the conclusions we draw from our data
analyses, and the need for better techniques to incorporate what might be divergent priors into
analyses. While their paper emphasizes each of these priors separately to illustrate the divergent
results, experts such as Wallerstein and Stephens may have equally valid insights into the research
problem, or may have pieces of the same picture. How should these differing views then be
reconciled?
20
5.1 Extending Western and Jackman
To illustrate the efficacy of the Dirichlet-process-based method I propose for aggregating priors,
I reassess the data used by Western and Jackman using a slightly larger body of hypothetical
priors. Rather than simply using the two paradigmatic examples of Stephens and Wallerstein,
I construct hypothetical priors for “experts” who may favor any of the explanatory variables
identified by Western and Jackman, as reasonable scholars of the literature may have a diverse
set of beliefs about key explanatory factors. For example, while Western and Jackman treat “left
government” as simply a control variable, my analysis assumes that some experts might consider
left governments to play a significant role in determining unionization. To reflect this diversity
of possible perspectives, I construct priors for 10 hypothetical experts, including Wallerstein and
Stephens. In addition to these two iconic scholars, I suppose that there is someone skeptical of
economic explanations—one who believes that government policies solely determine unioniza-
tion; two different camps of communist views, one governmentalist that believes a liberal state
can support unionization and another more classical, believing that only a large labor force could
solidify the aims of the proletariat and lead to organization; a neoliberal who believes that the
competition arising from economic concentration (industrialization) should decrease unioniza-
tion; two labor-supporting experts who believe that the labor force is most strongly determining
but who disagree about the magnitude; and two “uncertain” experts, who can agree on the same
explanatory variables as the other experts but who believe the magnitude of effects is small.
The prior means and precisions chosen for each of these hypothetical experts is reflected
in the figure below. The variances selected in the original Jackman and Western (1994) are often
quite large, and others are quite small, and I follow this convention in choosing variances cor-
responding to the priors of the new hypothetical experts. Precisions ( 1σ2 ) are shown for visual
clarity. As is evidenced in the figure, these prior means and precisions all seem like reasonable
positions that experts on the issue of unionization might hold, but at the same time they reflect
considerable diversity.
One approach to estimating a model with such diverse opinions of experts would generate
separate estimates, with each expert’s prior used in succession, and then compare results. This
framework is reasonable if we believe that while each prior is legitimate, only one provides
21
Economic Concentration Left Government Log Labor Force
Wallerstein
Stephens
Econ Skeptic
Communist 1
Communist 2
Neoliberal
Uncertain 1
Uncertain 2
Labor 1
Labor 2
-10 -5 0 5 10 0.00 0.25 0.50 0.75 -5 0 5 10Prior Mean
Expert Precision
High (>10)
Low (<10)
Figure 5: Hypothetical Expert Priors and Precisions
the “true” answer. As is especially true in studies of authoritarianism, where the information
environment is incredibly fragmented, however, one can also imagine a circumstance in which
each of these expert’s priors contains some element of the truth, but having an aggregate picture
would be preferable.
Current research in elicited priors recognizes this tradeoff to some extent. Because expert
priors are often elicited in focus group settings, rather than in separate sessions, however, this
problem is addressed by attempting to achieve a consensus prior, or trying to pool or average
the priors of the attending experts. This approach, I argue, loses considerable information, and a
Dirichlet-based method of aggregation should perform better, particularly when expert opinions
are divergent. To illustrate this with the Jackman and Western (1994) data, I estimate the model
from the original paper, and demonstrate the resulting posterior means and credible intervals
when the 10 hypothetical priors are averaged versus when they are clustered in a Dirichlet pro-
cess. I omit a comparison to pooling because the result would depend upon researcher-chosen
weights for expert opinions.
The results in Figure 6 demonstrate the ability of the Dirichlet process aggregation to better
incorporate a series of divergent priors into a common analysis of unionization, relative to aver-
aging priors. The intercepts across both models do not vary because the intercepts for all priors
22
Intercept Left Govt Log Labor Econ Concentration
Average
DP
0 20 40 0.0 0.1 0.2 0.3 -3e-04 -2e-04 -1e-04 0e+00 -5 0 5 10 15Posterior Mean and 95% Credible Interval
Model
Figure 6: Averaging vs. Dirichlet Process Prior Results
were set to be equal. The results do differ, however, in their estimation of the three explana-
tory variables. In estimating an effect for left governments, for example, the Dirichlet process
model recovers a coefficient similar to the high-precision prior means supplied by Wallerstein
and Stephens, who share a cluster, whereas the model that averages priors gives slightly more
weight to the views of communist and neoliberal experts. In particular, the averaging model may
be assigning more weight to the prior of “Communist 2,” who in the Dirichlet setup is in a cluster
by herself. A similar dynamic appears in the log labor force coefficients, which in both models
reflect the prior means that were near zero in most cases, but which differ in that greater weight
is assigned to the “Labor 1” and “Labor 2” experts in the Dirichlet process, who partially share a
cluster. Finally, while the coefficients for economic concentration appear the most extreme, their
confidence intervals do overlap. Even so, the Dirichlet process posterior mean centers on zero,
as is reflected in most of the prior means.
These results indicate that the Dirichlet process effectively handles the diversity of per-
spectives that experts may bring to an empirical analysis and reflect in their prior distributions.
At the same time, its results appear less volatile than those of the averaging model because the
Dirichlet process clustering is able to incorporate extreme views while identifying latent credi-
bility in the data structure that serves to diffuse less informative extreme perspectives. Even so,
23
the Dirichlet framework is very flexible: the standard setup provides equal weighting to clusters,
but a researcher themselves may have priors about the credibility or reliability of experts. The
researcher’s priors in turn can be instantiated as a hyperior, providing different weights for the
clusters created through the Dirichlet process. In this way, even with as few as 20 observations,
the information held by experts in the field, as well as researchers investigating current questions,
can be effectively leveraged to conduct data analyses.
6 Conclusions and Next Steps
An elicited priors approach, as the literature discussed above demonstrates, provides incredibly
useful additional insight in Bayesian analysis across a wide array of disciplines. In applying the
method within political science, however, examples are much more limited. As I have argued,
this is likely due to the fact that the current body of literature does not provide adequate guidance
for adapting elicited priors to social science settings. In particular, because “expertise” can be
a much broader and more nebulous concept in social scientific contexts, researchers are likely
to encounter greater diversity in the priors they elicit. Current techniques do not adequately
justify methods for aggregating these divergent opinions, nor do they offer concrete methods
for adjudicating the value of some priors versus others beyond the statistical accuracy of the
statements.
For researchers in the social sciences, and particularly political science, more work is
needed to be able to adapt an elicited priors approach to relevant research questions. This is
particularly the case for work that addresses authoritarian regimes or takes place in low-data
environments. These settings, often synonymous with small-n work and poor data or informa-
tion quality, are precisely the types of settings where a Bayesian approach should predominate
because of its ability to more easily handle modestly sized data. In these settings, however,
“experts” are likely to diverge more significantly in their opinions due to differing access to
information and biases resulting from political status. Accounting for these differences in an
elicited priors framework requires moving beyond the most common aggregation methods cur-
rently available (pooling and averaging).
In this paper, I have proposed a Dirichlet-based framework for addressing these diverging
priors, borrowing a technology commonly applied in text-as-data type analyses. This method has
24
greater flexibility and transparency, and can allow the researcher to aggregate priors according
to different latent categories of consensus without only relying on the priors’ statistical validity
in reference to the data. This method stands to be particularly useful in settings where small-n
and/or a complex modeling structure require informative priors, and where in principle experts
exist to offer prior information but they are an as-yet untapped resource. As the example with
the Jackman and Western (1994) data demonstrates, the Dirichlet-based approach competently
deals with the diversity of potential expert opinions, and facilitates estimating a model even with
sparse data.
The approach suggested here and the emphasis on eliciting priors may have its own down-
sides. In particular, elicitation of priors and the documentation of the research process in low-data
contexts is likely to be difficult and more time consuming than other approaches. The aim, how-
ever, is to generate an approach that would both facilitate the use of elicited priors more broadly,
while also particularly championing elicited priors as a tool for use in challenging research con-
texts where the lack of data or absence of identifiable expertise are particularly problematic.
This goal is especially well-suited to closing part of the quantitative/qualitative divide that has
emerged in some areas of political science, since the elicited priors approach would allow quan-
titative scholars to still conduct analyses while partnering with qualitative scholars whose deep
contextual knowledge could aid in the identification of experts and the elicitation process as well.
6.1 Additions and Extensions
To further the analysis included in this paper, I aim to conduct a series of simulations that will il-
lustrate the applications of the proposed method, and its performance relative to current methods
such as pooling and averaging. In particular, this will involve simulating both continuous and
discrete models/data with a series of generated priors “elicited from experts.” The simulation
will show how differing distributions of experts’ information (e.g., divergent versus convergent,
large versus small numbers of experts, large versus small number of data points elicited, etc.)
contribute to the estimation process with each of these methods for prior elicitation.
To guide this empirical assessment of the proposed model, however, I also aim to provide
a formal model of information elicitation. A general model of information elicitation is useful
as a baseline for developing expectations about the type of information we as researchers re-
25
ceive when using an elicited priors approach, especially in an authoritarian setting. In particular,
the authoritarian context for elicitation means that the information environment is constrained:
individuals have limited access to information and their understanding of situations and con-
sequences may be biased as a result. Whether and what information an “expert” offers during
elicitation is a product of this constrained information environment, and this condition in par-
ticular should shape how information from different “experts” is and can be combined to better
understand social processes at work. Constructing a formal model of why, whether, and what
information individuals are likely to offer during elicitation given authoritarian constraints can
provide more rigorous expectations for what methodology to employ when eliciting and aggre-
gating these priors.
References
Albert, Isabelle, Sophie Donnet, Chantal Guihenneuc-Jouyaux, Samantha Low-Choy, Kerrie Mengersen,
and Judith Rousseau. 2012. “Combining Expert Opinions in Prior Elicitation.” Bayesian Anal-
ysis 7 (3): 503–531.
Bakker, Ryan. 2009. “Re-measuring left-right: A comparison of SEM and Bayesian measurement
models for extracting left-right party placements.” Electoral Studies 28 (3): 413–421.
Blei, David, Andrew Y. Ng, and Michael I. Jordan. 2003. “Latent Dirichlet Allocation.” Journal of
Machine Learning Research 3:993–1022.
Daponte, Beth Osborne, Joseph B. Kadane, and Lara J. Wolfson. 1997. “Bayesian Demography:
Projecting the Iraqi Kurdish Population, 1977–1990.” Journal of the American Statistical Associ-
ation 92 (440): 1256–1267.
Datta, G.S., and M. Ghosh. 1991. “Asymptotic Optimality of Hierarchical Bayes Estimators and
Predictors.” Journal of Statistical Planning and Inference 29 (3): 229–243.
Dey, Dipak K., and Junfeng Liu. 2007. “A Quantitative Study of Quantile Based Direct Prior
Elicitation from Expert Opinion.” Bayesian Analysis 2 (1): 137–166.
Dey, D.K., and L.R. Birmiwal. 1994. “Robust Bayesian Analysis Using Divergence Measures.”
Statistics and Probability Letters 20 (4): 287–294.
26
Garthwaite, Paul H., Joseph B. Kadane, and Anthony O’Hagan. 2005. “Statistical Methods for
Eliciting Probability Distributions.” Journal of the American Statistical Association 100 (470):
680–700.
Gelman, Andrew. 2002. “Prior Distribution.” In Encyclopedia of Environmetrics, edited by Abdel H.
El-Shaarawi and Walter W. Piegorsch, 3:1634–1637. Chichester: John Wiley & Sons.
Gill, Jeff, and Lee D. Walker. 2005. “Elicited priors for Bayesian model specifications in political
science research.” Journal of Politics 67 (3): 841–872.
Gill, Jeff, and Christopher Witko. 2013. “Bayesian Analytical Methods: A Methodological Pre-
scription for Public Administration.” Journal of Public Administration Research and Theory 23
(2): 457–494.
Jackman, Simon, and Bruce Western. 1994. “Bayesian Inference for Comparative Research.” Amer-
ican Political Science Review 88 (2): 412–423.
Kadane, Joseph B., and Lara J. Wolfson. 1998. “Experiences in Elicitation.” Journal of the Royal
Statistical Society Series D (The Statistician) 47 (1): 3–19.
Kendall, Chad, Tommaso Nannicini, and Francesco Trebbi. 2015. “How Do Voters Respond to
Information? Evidence from a Randomized Campaign.” American Economic Review 105 (1):
322–353.
Lele, S.R., and A. Das. 2000. “Elicited data and incorporation of expert opinion for statistical
inference in spatial studies.” Mathematical Geology 32 (4): 465–487.
Lin, C.H., Y.C. Lin, and P. Raghubir. 2004. “The interaction between order of elicitation and event
controllability on the self-positivity bias.” In Advances in Consumer Research, Volume XXXI,
edited by B.E. Kahn and M.F. Luce, 31:523–529.
Morris, Rachel K., Gemma L. Malin, Elisabeth Quinlan-Jones, Lee J. Middleton, Karla Hemming,
Danielle Burke, Jane P. Daniels, Khalid S. Khan, Jon Deeks, and Mark D. Kilby. 2013. “Per-
cutaneous vesicoamniotic shunting versus conservative management for fetal lower urinary
tract obstruction PLUTO: a randomised trial.” Lancet 382 (9903): 1496–1506.
27
Neal, Radford M. 2000. “Markov Chain Sampling MMethod for Dirichlet Process Mixture Mod-
els.” Journal of Computational and Graphical Statistics 9 (2): 249–265.
O’Leary, Rebecca A., Samantha Low Choy, Justine V. Murray, Mary Kynn, Robert Denham, Tara
G. Martin, and Kerrie Mengersen. 2009. “Comparison of three expert elicitation methods for
logistic regression on predicting the presence of the threatened brush-tailed rock-wallaby
Petrogale penicillata.” Environmentrics 20 (4): 379–398.
Savage, Leonard J. 1971. “Elicitation of Personal Probabilities and Expectations.” Journal of the
American Statistical Association 66 (336): 783–801.
Shuford, Emir, and Thomas A. Brown. 1975. “Elicitation of Personal Probabilities and Their As-
sessment.” Instructional Science 4:137–188.
Sivaganesan, S. 1993. “Range of the Posterior Probability of an Interval for Priors with Unimodal-
ity Preserving Contaminations.” Annals of the Institute of Statistical Mathematics 45 (1): 187–
199.
Small, Mitchell J. 2008. “Methods for assessing uncertainty in fundamental assumptions and
associated models for cancer risk assessment.” Risk Analysis 28 (5): 1289–1307.
Spetzler, Carl S., and Carl-Axel S. Stael von Holstein. 1975. “Probability Encoding in Decision
Analysis.” Management Science 22:340–358.
Wang, Zhongming, and Hui Zhou. 2009. “A General Method of Prior Elicitation in Bayesian
Reliability Analysis.” In Proceedings of the 2009 8th International Conference on Reliability, Main-
tainability and Safety, Vols. I and II, edited by R. Kang, H.Z. Huang, M.J. Zuo, and Q. Miao,
415–419.
Wheeler, David C., Antonio Paez, Jamie Spinney, and Lance A. Waller. 2014. “A Bayesian ap-
proach to hedonic price analysis.” Papers in Regional Science 93 (3): 663–683.
Zaffalon, M. 2005. “Credible classification for environmental problems.” Environmental Modelling
and Software 20 (8): 1003–1012.
28