A Comparison of Two MCMC Algorithms for Hierarchical Mixture Models

Russell G. Almond⇤

Florida State University

Abstract

Mixture models form an important class ofmodels for unsupervised learning, allowingdata points to be assigned labels based ontheir values. However, standard mixturemodels procedures do not deal well with rarecomponents. For example, pause times instudent essays have di↵erent lengths depend-ing on what cognitive processes a studentengages in during the pause. However, in-stances of student planning (and hence verylong pauses) are rare, and thus it is dif-ficult to estimate those parameters from asingle student’s essays. A hierarchical mix-ture model eliminates some of those prob-lems, by pooling data across several of thehigher level units (in the example students)to estimate parameters of the mixture com-ponents. One way to estimate the parame-ters of a hierarchical mixture model is to useMCMC. But these models have several issuessuch as non-identifiability under label switch-ing that make them di�cult to estimate justusing o↵-the-shelf MCMC tools. This paperlooks at the steps necessary to estimate thesemodels using two popular MCMC packages:JAGS (random walk Metropolis algorithm)and Stan (Hamiltonian Monte Carlo). JAGS,Stan and R code to estimate the models andmodel fit statistics are published along withthe paper.

Key words: Mixture Models, Markov Chain MonteCarlo, JAGS, Stan, WAIC

⇤Paper presented at the Bayesian Application Workshop at Un-

certainty in Artificial Intelligence Conference 2014, Quebec City,

Canada.

1 Introduction

Mixture models (McLachlan & Peel, 2000) are a fre-quently used method of unsupervised learning. Theysort data points into clusters based on just their val-ues. One of the most frequently used mixture modelsis a mixture of normal distributions. Often the meanand variance of each cluster is learned along with theclassification of each data point.

As an example, Almond, Deane, Quinlan, Wagner, andSydorenko (2012) fit a mixture of lognormal distribu-tions to the pause time of students typing essays aspart of a pilot writing assessment. (Alternatively, thismodel can be described as a mixture of normals fit tothe log pause times.) Almond et al. found that mix-ture models seems to fit the data fairly well. The mix-ture components could correspond to di↵erent cogni-tive process used in writing (Deane, 2012) where eachcognitive process takes di↵erent amounts of time (i.e.,students pause longer when planning, than when sim-ply typing).

Mixture models are di�cult to fit because they dis-play a number of pathologies. One problem is com-ponent identification. Simply swapping the labels ofComponents 1 and 2 produces a model with identicallikelihoods. Even if a prior distribution is placed onthe mixture component parameters, the posterior ismultimodal. Second, it is easy to get a pathologicalsolution in which a mixture component consists of asingle point. These solutions are not desirable, andsome estimation tools constrain the minimum size ofa mixture component (Gruen & Leisch, 2008). Fur-thermore, if a separate variance is to be estimated foreach mixture component, several data points must beassigned to each component in order for the varianceestimate to have a reasonable standard error. There-fore, fitting a model with rare components requires alarge data size.

Almond et al. (2012) noted these limitations intheir conclusions. First, events corresponding to the

1

highest-level planning components in Deane (2012)’scognitive model would be relatively rare, and hencewould lumped in with other mixture components dueto size restrictions. Second, some linguistic contexts(e.g., between Sentence pauses) were rare enough thatfitting a separate mixture model to each student wouldnot be feasible.

One work-around is a hierarchical mixture model. Aswith all hierarchical models, it requires units at twodi↵erent levels (in this example, students or essaysare Level 2 and individual pauses are Level 1). Theassumption behind the hierarchical mixture model isthat the mixture components will look similar acrossthe second level units. Thus, the mean and variance ofMixture Component 1 for Student 1 will look similarto those for Student 2. Li (2013) tried this on some ofthe writing data.

One problem that frequently arises in estimating mix-ture models is determining how many mixture compo-nents to use. What is commonly done is to estimatemodels for K = 2, 3, 4, . . . up to some small maximumnumber of components (depending on the size of thedata). Then a measure of model–data fit, such as AIC,DIC or WAIC (see Gelman et al., 2013, Chapter 7), iscalculated for each model and the model with the bestfit index is chosen. These methods look at the deviance(minus twice the log likelihood of the data) and adjustit with a penalty for model complexity. Both DIC andWAIC require Markov chain Monte Carlo (MCMC) tocompute, and require some custom coding for mixturemodels because of the component identification issue.

This paper is a tutorial for replicating the method usedby Li (2013). The paper walks through a script writ-ten in the R language (R Core Team, 2014) which per-forms most of the steps. The actual estimation is doneusing MCMC using either Stan (Stan DevelopmentTeam, 2013) or JAGS (Plummer, 2012). The R scriptsalong with the Stan and JAGS models and some sam-ple data are available at http://pluto.coe.fsu.edu/mcmc-hierMM/.

2 Mixture Models

Let i 2 {1, . . . , I} be a set of indexes over the sec-ond level units (students in the example) and letj 2 {1, . . . , Ji} be the first level units (pause eventsin the example). A hierarchical mixture model is byadding a Level 2 (across student) distribution over theparameters of the Level 1 (within student) mixturemodel. Section 2.1 describes the base Level 1 mixturemodel, and Section 2.2 describes the Level 2 model.Often MCMC requires reparameterization to achievebetter mixing (Section 2.3). Also, there are certain pa-rameter values which result in infinite likelihoods. Sec-

Figure 1: Non-hierarchical Mixture Model

tion 2.4 describes prior distributions and constraintson parameters which keep the Markov chain away fromthose points.

2.1 Mixture of Normals

Consider a collection observations, Yi =(Yi,1, . . . , Yi,Ji) for a single student, i. Assumethat the process that generated these data is a mix-ture of K normals. Let Zi,j ⇠ cat(⇡i) be a categoricallatent index variable indicating which component Ob-servation j comes from and let Y

⇤i,j,k ⇠ N (µi,k,�i,k)

be the value of Yi,j which would be realized whenZi,j = k.

Figure 1 shows this model graphically. The plates in-dicate replication over categories (k), Level 1 (pauses,j) and Level 2 (students, i) units. Note that there isno connection across plates, so the model fit to eachLevel 2 unit is independent of all the other Level 2units. This is what Gelman et al. (2013) call the nopooling case.

The latent variables Z and Y

⇤ can be removed fromthe likelihood for Yi,j by summing over the possiblevalues of Z. The likelihood for one student’s data, Yi,is

Li(Yi|⇡i,µi,�i) =JiY

j=1

KX

k=1

⇡i,k�(Yi,j � µi,k

�i,k) (1)

where �(·) is the unit normal density.

Although conceptually simple, there are a number ofissues that mixture models can have. The first is-sue is that the component labels cannot be identifiedfrom data. Consider the case with two components.The model created by swapping the labels for Compo-nents 1 and 2 with new parameters ⇡0

i = (⇡i,2,⇡i,1),µ0

i = (µi,2, µi,1), and �0i = (�i,2,�i,1) has an identical

2

likelihood. For the K component model, any permu-tation of the component labels produces a model withidentical likelihood. Consequently, the likelihood sur-face is multimodal.

A common solution to the problem is to identify thecomponents by placing an ordering constraint on oneof the three parameter vectors: ⇡, µ or �. Section 4.1returns to this issue in practice.

A second issue involves degenerate solutions whichcontain only a single data point. If a mixture com-ponent has only a single data point, its standard devi-ation, �i,k will go to zero, and ⇡i,k will approach 1/Ji,which is close to zero for large Level 1 data sets. Notethat if either ⇡i,k ! 0 or �i,k� > 0 for any k, then thelikelihood will become singular.

Estimating �i,k requires a minimum number of datapoints from Component k (⇡i,kJi > 5 is a rough mini-mum). If ⇡i,k is believed to be small for some k, then alarge (Level 1) sample is needed. As K increases, thesmallest value of ⇡i,k becomes smaller so the minimumsample size increases.

2.2 Hierarchical mixtures

Gelman et al. (2013) explains the concept of a hier-archical model using a SAT coaching experiment thattook place at 8 di↵erent schools. Let Xi ⇠ N (µi,�i)be the observed e↵ect at each school, where the schoolspecific standard error �i is known (based mainly onthe sample size used at that school). There are threeways to approach this problem: (1) No pooling. Es-timate µi separately with no assumption about aboutthe similarity of µ across schools. (2) Complete pool-ing. Set µi = µ

0

for all i and estimate µ

0

(3) Par-tial pooling. Let µi ⇠ N (µ

0

, ⌫) and now jointly es-timate µ

0

, µ

1

, . . . , µ

8

. The no pooling approach pro-duces unbiased estimates for each school, but it hasthe largest standard errors, especially for the smallestschools. The complete pooling approach ignores theschool level variability, but has much smaller standarderrors. In the partial pooling approach, the individualschool estimates are shrunk towards the grand mean,µ

0

, with the amount of shrinkage related to the size ofthe ratio ⌫

�2

/(⌫�2+�

�2

i ); in particular, there is moreshrinkage for the schools which were less precisely mea-sured. Note that the higher level standard deviation,⌫ controls the amount of shrinkage: the smaller ⌫ isthe more the individual school estimates are pulled to-wards the grand mean. At the limits, if ⌫ = 0, the par-tial pooling model is the same as the complete poolingmodel and if ⌫ = 1 then the partial pooling model isthe same as the no pooling model.

Li (2013) builds a hierarchical mixture model forthe essay pause data. Figure 1 shows the no pool-

Figure 2: Hierarchical Mixture Model

ing mixture model. To make the hierarchical model,add across-student prior distributions for the student-specific parameters parameters, ⇡i, µi and �i. Be-cause JAGS parameterizes the normal distributionwith precisions (reciprocal variances) rather than stan-dard deviations, ⌧ i are substituted for the standarddeviations �i. Figure 2 shows the hierarchical model.

Completing the model requires specifying distributionsfor the three Level 1 (student-specific) parameters. Inparticular, let

⇡i = (⇡i,1, . . . ,⇡i,K) ⇠ Dirichlet(↵1

, . . . ,↵k) (2)

µi,k ⇠ N (µ0,k,�0,k) (3)

log(⌧i,k) ⇠ N (log(⌧0,k), �0,k) (4)

This introduces new Level 2 (across student) param-eters: ↵, µ

0

, �0

, ⌧0

, and �0

. The likelihood for asingle student, Li(Yi|⇡i,µi, ⌧ i) is given (with a suit-able chance of variable) by Equation 1. To get thecomplete data likelihood, multiply across the studentsunits (or to avoid numeric overflow, sum the log likeli-hoods). If we let ⌦ be the complete parameter (⇡i,µi,⌧ i for each student, plus ↵, µ

0

, �0

, ⌧0

, and �0

), then

L(Y|⌦) =IX

i=1

logLi(Yi|⇡i,µi, ⌧ i) . (5)

Hierarchical models have their own pathologies whichrequire care during estimation. If either of the stan-dard deviation parameters, �

0,k or �0,k, gets too close

to zero or infinity, then this could cause the log poste-rior to go to infinity. These cases correspond to the nopooling and complete pooling extremes of the hierar-chical model. Similarly, the variance of the Dirichletdistribution is determined by ↵N =

PKk=1

↵k. If ↵N

is too close to zero, this produces no pooling in theestimates of ⇡i and if ↵N is too large, then there is

3

nearly complete pooling in those estimates. Again,those values of ↵N can cause the log posterior to be-come infinite.

2.3 Reparameterization

It is often worthwhile to reparameterize a model in or-der to make MCMC estimation more e�cient. Bothrandom walk Metropolis (Section 3.2) and Hamilto-nian Monte Carlo (Section 3.3) work by moving a pointaround the parameter space of the model. The geom-etry of that space will influence how fast the Markovchain mixes, that is, moves around the space. If thegeometry is unfavorable, the point will move slowlythrough the space and the autocorrelation will be high(Neal, 2011). In this case a very large Monte Carlosample will be required to obtain reasonable MonteCarlo error for the parameter estimates.

Consider once more the Eight Schools problem whereµi ⇠ N (µ

0

, ⌫). Assume that we have a sampler thatworks by updating the value of the µi’s one at a timeand then updating the values of µ

0

and ⌫. When up-dating µi, if ⌫ is small then values of µi close to µ

0

willhave the highest conditional probability. When updat-ing ⌫ if all of the values of µi are close to µ

0

, then smallvalues of ⌫ will have the highest conditional probabil-ity. The net result is a chain in which the movementfrom states with small ⌫ and the µi’s close togetherto states with large ⌫ and µi’s far apart takes manysteps.

A simple trick produces a chain which moves muchmore quickly. Introduce a series of auxiliary variables✓i ⇠ N (0, 1) and set µi = µ

0

+ ⌫✓i. Note that themarginal distribution of µi has not changed, but thegeometry of the ✓, µ

0

, ⌫ space is di↵erent from the ge-ometry of the µ, µ

0

, ⌫ space, resulting in a chain thatmoves much more quickly.

A similar trick works for modeling the relationshipsbetween ↵ and ⇡i. Setting ↵k = ↵

0,k ⇤ ↵N , where↵

0

has a Dirichlet distribution and ↵N has a gammadistribution seems to work well for the parameters ↵ ofthe Dirichlet distribution. In this case the parametershave a particularly easy to interpret meaning: ↵

0

isthe expected value of the Dirichlet distribution and ↵N

is the e↵ective sample size of the Dirichlet distribution.

Applying these two tricks to the parameters of thehierarchical model yields the following augmented pa-rameterization.

↵k = ↵

0,k ⇤ ↵N (6)

µi,k = µ

0,k + ✓i,k�0,k (7)

✓i,k ⇠ N (0, 1) (8)

log(⌧i,k) = log(⌧0,k) + ⌘i,k�0,k (9)

⌘i,k ⇠ N (0, 1) (10)

2.4 Prior distributions and parameterconstraints

Rarely does an analyst have enough prior informa-tion to completely specify a prior distribution. Usu-ally, the analyst chooses a convenient functional formand chooses the hyperparameters of that form basedon available prior information (if any). One popularchoice is the conjugate distribution of the likelihood.For example, if the prior for a multinomial probability,⇡i follows a Dirichlet distribution with hyperparame-ter, ↵, then the posterior distribution will also be aDirichlet with a hyperparameter equal to the sum ofthe prior hyperparameter and the data. This gives thehyperparameter of a Dirichlet prior a convenient in-terpretation as pseudo-data. A convenient functionalform is one based on a normal distribution (sometimeson a transformed parameter, such as the log of theprecision) whose mean can be set to a likely value ofthe parameter and whose standard deviation can beset so that all of the likely values for the parameterare within two standard deviations of the mean. Notethat for location parameters, the normal distributionis often a conjugate prior distribution.

Proper prior distributions are also useful for keepingparameter values away from degenerate or impossiblesolutions. For example, priors for standard deviations,�

0,k and �

0,k must be strictly positive. Also, the groupprecisions for each student, ⌧ i, must be strictly pos-itive. The natural conjugate distribution for a preci-sion is a gamma distribution, which is strictly positive.The lognormal distribution, used in Equations 4 and 9,has a similar shape, but its parameters can be inter-preted as a mean and standard deviation on the logscale. The mixing probabilities ⇡i must be definedover the unit simplex (i.e., they must all be between 0and 1 and they must sum to 1), as must the expectedmixing probabilities ↵

0

; the Dirichlet distribution sat-isfies this constraint and is also a natural conjugate.Finally, ↵N must be strictly positive; the choice of thechi-squared distribution as a prior ensures this.

There are other softer restraints on the parameters. If�

0,k or �

0,k gets too high or too low for any value ofk, the result is a no pooling or complete pooling so-lution on that mixture component. For both of theseparameters (.01, 100) seems like a plausible range. A

4

lognormal distribution which puts the bulk of its prob-ability mass in that range should keep the model awayfrom those extremes. Values of ⌧i,k that are too smallrepresent the collapse of a mixture component ontoa single point. Again, if ⌧

0,k is mostly in the range(.01, 100) the chance of an extreme value of ⌧i,k shouldbe small. This yields the following priors:

log(�0k) ⇠ N (0, 1) (11)

log(�0k) ⇠ N (0, 1) (12)

log(⌧0k) ⇠ N (0, 1) (13)

High values of ↵N also correspond to a degenerate so-lution. In general, the gamma distribution has aboutthe right shape for the prior distribution of ↵N , and weexpect it to be about the same size as I, the numberof Level-2 units. The choice of prior is a chi-squareddistribution with 2 ⇤ I degrees of freedom.

↵N ⇠ �

2(I ⇤ 2) (14)

The two remaining parameters we don’t need to con-strain too much. For µ

0,k we use a di↵use normalprior (one with a high standard deviation), and for ↵

0

we use a Je↵rey’s prior (uniform on the logistic scale)which is the Dirichlet distribution with all values setto 1/2.

µ

0,k ⇠ N(0, 1000) (15)

↵0

⇠ Dirichlet(0.5, . . . , 0.5) (16)

The informative priors above are not su�cient to al-ways keep the Markov chain out of trouble. In partic-ular, it can still reach places where the log posteriordistribution is infinite. There are two di↵erent placeswhere these seem to occur. One is associated with highvalues of ↵N . Putting a hard limit of ↵N < 500 seemsto avoid this problem (when I = 100). Another possi-ble degenerate spot is when ↵k ⇡ 0 for some k. Thismeans that ⇡i,k will be essentially zero for all students.Adding .01 to all of the ↵k values in Equation 2 seemsto fix this problem.

⇡i = (⇡i,1, . . . ,⇡i,K) ⇠ Dirichlet(↵1

+.01, . . . ,↵k+.01)

3 Estimation Algorithms

There are generally two classes of algorithms usedwith both hierarchical and mixture models. The ex-pectation maximization (EM) algorithm (Section 3.1)searches for a set of parameter values that maximizesthe log posterior distribution of the data (or if the priordistribution is flat, it maximizes the log likelihood). It

comes up with a single parameter estimate but doesnot explore the entire space of the distribution. In con-trast, MCMC algorithms explore the entire posteriordistribution by taking samples from the space of possi-ble values. Although the MCMC samples are not inde-pendent, if the sample is large enough it converges indistribution to the desired posterior. Two approachesto MCMC estimation are the random walk Metropolisalgorithm (RWM; used by JAGS, Section 3.2) and theHamiltonian Monte Carlo algorithm (HMC; used byStan, Section 3.3).

3.1 EM Algorithm

McLachlan and Krishnan (2008) provides a review ofthe EM algorithm with emphasis on mixture models.The form of the EM algorithm is particularly simplefor the special case of a non-hierarchical mixture ofnormals. It alternates between an E-step where thep(Zi,j = k) = pi,j,k is calculated for every observation,j, and every component, k and an M-step where themaximum likelihood values for ⇡i,k, µi,k and �i,k arefound by taking moments of the data set Yi weightedby the component probabilities, pi,j,k.

A number of di↵erent problems can crop up with usingEM to fit mixture models. In particular, if ⇡i,k goes tozero for any k, that component essentially disappearsfrom the mixture. Also, if �i,k goes to zero the mix-ture component concentrates on a single point. Fur-thermore, if µi,k = µi,k0 and �i,k = �i,k0 for any pairof components the result is a degenerate solution withK � 1 components.

As the posterior distribution for the mixture model ismultimodal, the EM algorithm only finds a local max-imum. Running it from multiple starting points mayhelp find the global maximum; however, in practice itis typically run once. If the order of the componentsis important, the components are typically relabeledafter fitting the model according to a predefined rule(e.g., increasing values of µi,k).

Two packages are available for fitting non-hierarchicalmixture models using the EM algorithm in R (R CoreTeam, 2014): FlexMix (Gruen & Leisch, 2008) andmixtools (Benaglia, Chauveau, Hunter, & Young,2009). These two packages take di↵erent approachesto how they deal with degenerate solutions. FlexMix

will combine two mixture components if they get tooclose together or the probability of one component getstoo small (by default, if ⇡i,k < .05). Mixtools, on theother hand, retries from a di↵erent starting point whenthe the EM algorithm converges on a degenerate so-lution. If it exceeds the allowed number of retries, itgives up.

Neither mixtools nor FlexMix provides standard er-

5

rors for the parameter estimates. The mixtools pack-age recommends using the bootstrap (resampling fromthe data distribution) to calculate standard errors, andprovides a function to facilitate this.

3.2 Random-walk Metropolis Algorithm(RWM; used by JAGS)

Geyer (2011) gives a tutorial summary of MCMC. Thebasic idea is that a mechanism is found for construct-ing a Markov chain whose stationary distribution is thedesired posterior distribution. The chain is run untilthe analyst is reasonably sure it has reach the station-ary distribution (these early draws are discarded asburn-in). Then the the chain is run some more until itis believed to have mixed throughout the entire poste-rior distribution. At this point it has reached pseudo-convergence (Geyer calls this pseudo-convergence, be-cause without running the chain for an infinite lengthof time, there is no way of telling if some part of theparameter space was never reached.) At this pointthe mean and standard error of the parameters areestimated from the the observed mean and standarddeviation of the parameter values in the MCMC sam-ple.

There are two sources of error in estimates made fromthe MCMC sample. The first arises because the ob-served data are a sample from the universe of potentialobservations. This sampling error would be presenteven if the posterior distribution could be computedexactly. The second is the Monte Carlo error thatcomes from the estimation of the posterior distribu-tion with the Monte Carlo sample. Because the drawsfrom the Markov chain are not statistically indepen-dent, this Monte Carlo error does not fall at the rateof 1/

pR (where R is the number of Monte Carlo sam-

ples). It is also related to the autocorrelation (corre-lation between successive draws) of the Markov chain.The higher the autocorrelation, the lower the e↵ectivesample size of the Monte Carlo sample, and the higherthe Monte Carlo error.

Most methods for building the Markov chain are basedon the Metropolis algorithm (see Geyer, 2011, for de-tails). A new value for one or more parameter is pro-posed and the new value is accepted or rejected ran-domly according to the ratio of the posterior distribu-tion and the old and new points, with a correction fac-tor for the mechanism used to generate the new sam-ple. This is called a Metropolis or Metropolis-Hastingsupdate (the latter contains a correction for asymmet-ric proposal distributions). Gibbs sampling is a specialcase in which the proposal is chosen in such a way thatit will always be accepted.

As the form of the proposal distribution does not mat-

ter for the correctness of the algorithm, the most com-mon method is to go one parameter at a time and adda random o↵set (a step) to its value, accepting or re-jecting it according to the Metropolis rule. As thisdistribution is essentially a random walk over the pa-rameter space, this implementation of MCMC is calledrandom walk Metropolis (RWM). The step size is acritical tuning parameter. If the average step size istoo large, the value will be rejected nearly every cycleand the autocorrelation will be high. If the step sizeis too small, the chain will move very slowly throughthe space and the autocorrelation will be high. Gibbssampling, where the step is chosen using a conjugatedistribution so the Metropolis-Hastings ratio alwaysaccepts, is not necessarily better. Often the e↵ectivestep size of the Gibbs sampler is small resulting in highautocorrelation.

Most packages that use RWM do some adaptation onthe step size, trying to get an optimal rejection rate.During this adaptation phase, the Markov chain doesnot have the correct stationary distribution, so thoseobservations must be discarded, and a certain amountof burn-in is needed after the adaptation finishes.

MCMC and the RWM algorithm were made popularby their convenient implementation in the BUGS soft-ware package (Thomas, Spiegelhalter, & Gilks, 1992).With BUGS, the analyst can write down the modelin a form very similar to the series of equations usedto describe the model in Section 2, with a syntax de-rived from the R language. BUGS then compiles themodel into pseudo-code which produces the Markovchain, choosing to do Gibbs sampling or random walkMetropolis for each parameter depending on whetheror not a convenient conjugate proposal distributionwas available. The output could be exported in aform that could be read by R, and the R package coda(Plummer, Best, Cowles, & Vines, 2006) could be usedto process the output. (Later, WinBUGS would buildsome of that output processing into BUGS.)

Although BUGS played an important role in encourag-ing data analysts to use MCMC, it is no longer activelysupported. This means that the latest developmentsand improvements in MCMC do not get incorporatedinto its code. Rather than use BUGS, analysts areadvised to use one of the two successor software pack-ages: OpenBUGS (Thomas, O’Hara, Ligges, & Sturtz,2006) or JAGS (just another Gibbs sampler; Plummer,2012). The R package rjags allows JAGS to be calledfrom R, and hence allows R to be used as a scriptinglanguage for JAGS, which is important for serious an-alytic e↵orts. (Similar packages exist for BUGS andOpenBUGS.)

6

3.3 Hamiltonian Monte Carlo (HMC; usedby Stan)

Hamiltonian Monte Carlo (HMC) (Neal, 2011) is avariant on the Metropolis Algorithm which uses a dif-ferent proposal distribution than RWM. In HMC, thecurrent draw from the posterior is imagined to be asmall particle on a hilly surface (the posterior distri-bution). The particle is given a random velocity and isallowed to move for several discrete steps in that direc-tion. The movement follows the laws of physics, so theparticle gains speed when it falls down hills and losesspeed when it climbs back up the hills. In this mannera proposal is generated that can be a great distancefrom the original starting point. The proposed point isthen accepted or rejected according to the Metropolisrule.

The software package Stan (Stan Development Team,2013) provides support for HMC. As with BUGS andJAGS, the model of Section 2 is written in pseudo-code, although this time the syntax looks more likeC++ than R. Rather than translate the model intointerpreted code, Stan translates it into C++ thencompiles and links it with existing Stan code to runthe sampler. This has an initial overhead for the com-pilation, but afterwards, each cycle of the sampler runsfaster. Also, as HMC generally has lower autocorrela-tion than random walk Metropolis, smaller run lengthscan be used, making Stan considerably faster thanJAGS in some applications. A package rstan is avail-able to link Stan to R, but because of the compilationstep, it requires that the user have the proper R de-velopment environment set up.

HMC has more tuning parameters than random walkMetropolis: the mass of the particle, the distributionof velocities and the number of steps to take in eachdirection must be selected to set up the algorithm.Stan uses a warm-up phase to do this adaptation. Therecommended procedure is to use approximately 1/2the samples for warm-up as a longer warm-up produceslower autocorrelations when actively sampling.

Stan has some interesting features that are not presentin BUGS or JAGS. For one, it does not require everyparameter to have a proper prior distribution (as longas the posterior is proper). It will simply put a uni-form prior over the space of possible values for anyparameter not given a prior distribution. However,using explicit priors has some advantages for the ap-plication to student pause data. In particular, whendata for a new student become available, the poste-rior parameters for the previous run can be input intoStan (or JAGS) and the original calibration model canbe reused to estimate the parameters for new student(Mislevy, Almond, Yan, & Steinberg, 1999).

3.4 Parallel Computing and Memory Issues

As most computers have multiple processors, paral-lel computing can be used to speed up MCMC runs.Often multiple Markov chains are run and the resultsare compared to assess pseudo-convergence and thencombined for inference (Gelman & Shirley, 2011). Thisis straightforward using the output processing packagecoda. It is a little bit trickier using the rstan package,because many of the graphics require a full stanfitobject. However, the conversion from Stan to codaformat for the MCMC samples is straightforward.

In the case of hierarchical mixture models, there isan even easier way to take advantage of multiple pro-cesses. If the number of components, K, is unknown,the usual procedure is to take several runs with dif-ferent values of K and compare the fit. Therefore, if4 processors were available, one could run all of thechains for K = 2, one for K = 3, and one for K = 4,leaving one free to handle operating system tasks.

In most modern computers, the bottleneck is usuallynot available CPU cycles, but available memory. Forrunning 3 chains with I = 100 students and R = 5000MCMC samples in each chain, the MCMC sample cantake up to 0.5GB of memory! Consequently, it is crit-ically important to monitor memory usage when run-ning multiple MCMC runs. If the computer starts re-quiring swap (disk-based memory) in addition to phys-ical memory, then running fewer processes will proba-bly speed up the computations.

Another potential problem occurs when storing the re-sult of each run between R sessions in the .RData file.Note that R attempts to load the entire contents ofthat data file into memory when starting R. If thereare the results of several MCMC runs in there, the.RData file can grow to several GB in size, and R cantake several minutes to load. (In case this problemarises, it is recommended that you take a make a copyof the .RData after the data have been cleaned and allthe auxiliary functions loaded but before starting theMCMC runs, and put it someplace safe. If the .RDatafile gets to be too large it can be simply replaced withthe backup.)

In order to prevent the .RData file from growingunmanageably large, it recommended that the workspace not be saved at the end of each run. Instead,run a block of code like this

assign(runName,result1)

outfile <-

gzfile(paste(runName,"R","gz",sep="."),

open="wt")

dump(runName,file=outfile)

close(outfile)

7

after all computations are completed. Here result1

is a variable that gathers together the portion of theresults to keep, and runName is a character string thatprovides a unique name for the run.

Assume that all of the commands necessary to performthe MCMC analysis are in a file script.R. To run thisfrom a command shell use the command:

R CMD BATCH --slave script.R

This will run the R code in the script, using the .RDatafile in the current working directory, and put the out-put into script.Rout. The --slave switch performstwo useful functions. First, it does not save the .RDatafile at the end of the run (avoiding potential memoryproblems). Second, it suppresses echoing the script tothe output file, making the output file easier to read.Under Linux and Mac OS X, the command

nohup R CMD BATCH --slave script.R &

runs R in the background. The user can log out of thecomputer, but as long as the computer remains on, theprocess will continue to run.

4 Model Estimation

A MCMC analysis always follows a number of similarsteps, although the hierarhical mixture model requiresa couple of additional steps. Usually, it is best to writethese as a script because the analysis will often needto be repeated several times. The steps are as follows:

1. Set up parameters for the run. This includeswhich data to analyze, how many mixture compo-nents are required (i.e., K), how long to run theMarkov chain, how many chains to run, what theprior hyperparameters are.

2. Clean the data. In the case of hierarchical mixturemodels, student data vectors which are too shortshould be eliminated. Stan does not accept miss-ing values, so NAs in the data need to be replacedwith a finite value. For both JAGS and Stan thedata are bundled with the prior hyperparametersto be passed to the MCMC software.

3. Set initial values. Initial values need to be chosenfor each Markov chain (see Section 4.2).

4. Run the Markov Chain. For JAGS, this consistsof four substeps: (a) run the chain in adaptationmode, (b) run the chain in normal mode for theburn-in, (c) set up monitors on the desired param-eters, and (d) run the chain to collect the MCMCsample. For Stan, the compilation, warm-up andsampling are all done in a single step.

5. Identify the mixture components. Ideally, theMarkov chains have visited all of the modes of theposterior distribution, including the ones whichdi↵er only by a permutation of the component la-bels. Section 4.1 describes how to permute thecomponent labels are permuted so that the com-ponent labels are the same in each MCMC cycle.

6. Check pseudo-convergence. Several statistics andplots are calculated to see if the chains havereached pseudo-convergence and the sample sizeis adequate. If the sample is inadequate, then ad-ditional samples are collected (Section 4.3).

7. Draw Inferences. Summaries of the posterior dis-tribution for the the parameters of interest arecomputed and reported. Note that JAGS o↵erssome possibilities here that Stan does not. In par-ticular, JAGS can monitor just the cross-studentparameters (↵

0

, ↵N , µ0

, �0

, log(⌧0

), and �0

) fora much longer time to check pseudo-convergence,and then a short additional run can be used todraw inferences about the student specific param-eters, ⇡i, µi and ⌧ i (for a considerable memorysavings).

8. Data point labeling. In mixture models, it is some-times of interest to identify which mixture com-ponent each observation Yi,j comes from (Sec-tion 4.5).

9. Calculate model fit index. If the goal is to comparemodels for several di↵erent values of K, then ameasure of model fit such as DIC or WAIC shouldbe calculated (Section 5).

4.1 Component Identification

The multiple modes in the posterior distribution formixture models present a special problem for MCMC.In particular, it is possible for a Markov chain to getstuck in a mode corresponding to a single componentlabeling and never mix to the other component la-belings. (This especially problematic when coupledwith the common practice of starting several paral-lel Markov chains.) Fruthwirth-Schnatter (2001) de-scribes this problem in detail.

One solution is to constrain the parameter space to fol-low some canonical ordering (e.g., µi,1 µi,2 · · · µi,K). Stan allows parameter vectors to be specifiedas ordered, that is restricted to be in increasing order.This seems tailor-made for the identification issue. Ifan order based on µ

0

is desired, the declaration:

ordered[K] mu0;

enforces the ordering constraint in the MCMC sam-pler. JAGS contains a sort function which can achieve

8

a similar e↵ect. Fruthwirth-Schnatter (2001) rec-ommends against this solution because the resultingMarkov chains often mix slowly. Some experimenta-tion with the ordered restriction in Stan confirmed thisfinding; the MCMC runs took longer and did not reachpseudo-convergence as quickly when the ordered re-striction was not used.

Fruthwirth-Schnatter (2001) instead recommends let-ting the Markov chains mix across all of the possi-ble modes, and then sorting the values according tothe desired identification constraints post hoc. JAGSprovides special support for this approach, o↵ering aspecial distribution dnormmix for mixtures of normals.This uses a specially chosen proposal distribution toencourage jumping between the multiple modes in theposterior. The current version of Stan does not pro-vide a similar feature.

One result of this procedure is that the MCMC samplewill have a variety of orderings of the components; eachdraw could potentially have a di↵erent labeling. Forexample, the MCMC sample for the parameter µi,1

will in fact be a mix of samples from µi,1, . . . , µi,K .The average of this sample is not likely to be a goodestimate of µi,1. Similar problems arise when lookingat pseudo-convergence statistics which are related tothe variances of the parameters.

To fix this problem, the component labels need to bepermuted separately for each cycle of the MCMC sam-ple. With a one-level mixture model, it is possibleto identify the components by sorting on one of thestudent-level parameters, ⇡i,k, µi,k or ⌧i,k. For thehierarchical mixture model, one of the cross-studentparameters, ↵

0,k, µ0,k, or ⌧

0,k, should be used. Notethat in the hierarchical models some of the student-level parameters might not obey the constraints. Inthe case of the pause time analysis, this is acceptableas some students may behave di↵erently from most oftheir peers (the ultimate goal of the mixture modelingis to identify students who may benefit from di↵erentapproaches to instruction). Choosing an identificationrule involves picking which parameter should be usedfor identification at Level 2 (cross-student) and usingthe corresponding parameter is used for student-levelparameter identification.

Component identification is straightforward. Let ! bethe parameter chosen for model identification. Fig-ure 3 describes the algorithm.

4.2 Starting Values

Although the Markov chain should eventually reachits stationary distribution no matter where it starts,starting places that are closer to the center of the dis-tribution are better in the sense that the chain should

for each Markov chain c: dofor each MCMC sample r in Chain c: doFind a permutation of indexes k

01

, . . . , k

0K so

that !c,r,k01 · · · !c,r,k0

1.

for ⇠ in the Level 2 parameters{↵

0

,µ0

,�0

, ⌧0

,�0

}: doReplace ⇠c,r with (⇠c,r,k0

1, . . . , ⇠c,r,k0

K).

end for{Level 2 parameter}if inferences about Level 1 parameters are de-sired thenfor ⇠ in the Level 1 parameters {⇡,µ, ⌧ , }do

for each Level 2 unit (student), i doReplace ⇠c,r,i with(⇠c,r,i,k0

1, . . . , ⇠c,r,i,k0

K).

end for{Level 1 unit}end for{Level 1 parameter}

end ifend for{Cycle}

end for{Markov Chain}

Figure 3: Component Identification Algorithm

reach convergence more quickly. Gelman et al. (2013)recommend starting at the maximum likelihood esti-mate when that is available. Fortunately, o↵-the shelfsoftware is available for finding the maximum likeli-hood estimate of the individual student mixture mod-els. These estimates can be combined to find startingvalues for the cross-student parameters. The initialvalues can be set by the following steps:

1. Fit a separate mixture model for students usingmaximum likelihood. The package mixtools isslightly better for this purpose than FlexMix asit will try multiple times to get a solution withthe requested number of components. If the EMalgorithm does not converge in several tries, setthe values of the parameters for that student toNA and move on.

2. Identify Components. Sort the student-level pa-rameters, ⇡i, µi, �i and ⌧ i according to the de-sired criteria (see Section 4.1.

3. Calculate the Level 2 initial values. Most of thecross-student parameters are either means or vari-ances of the student-level parameters. The initialvalue for ↵

0

is the mean of ⇡i across the studentsi (ignoring NAs). The initial values for µ

0

and �0

are the mean and standard deviation of µi. Theinitial values for log(⌧

0

) and �0

are the mean andstandard deviation of log(⌧ i).

4. Impute starting values for maximum likelihood es-timates that did not converge. These are the NAs

9

from Step 1. For each student i that did notconverge in Step 1, set ⇡i = ↵

0

, µi = µ0

and⌧ i = ⌧

0

.

5. Compute initial values for ✓i and ⌘i. These canbe computed by solving Equations 7 and 9 for ✓i

and ⌘i.

6. Set the initial value of ↵N = I. Only one param-eter was not given an initial value in the previoussteps, that is the e↵ective sample size for ↵. Setthis equal to the number of Level 2 units, I. Theinitial values of ↵ can now be calculated as well.

This produces a set of initial values for a single chain.Common practice for checking for pseudo-convergenceinvolves running multiple chains and seeing if they ar-rive the same stationary distribution. Di↵erent start-ing values for the second and subsequent chains canbe found by sampling some fraction � of the Level 1units from each Level 2 unit. When Ji is large, � = .5seems to work well. When Ji is small, � = .8 seemsto work better. An alternative would be to take abootstrap sample (a new sample of size Ji drawn withreplacement).

4.3 Automated Convergence Criteria

Gelman and Shirley (2011) describe recommendedpractice for assessing pseudo-convergence of a Markovchain. The most common technique is to run severalMarkov chains and look at the ratio of within-chainvariance to between-chain variance. This ratio is calledbR, and it comes in both a univariate (one parameterat a time) and a multivariate version. It is natural tolook at parameters with the same name together (e.g.,µ

0

, �0

, ⌧0

, and �0

). Using the multivariate version ofbR with ↵

0

and ⇡i requires some care because calcu-lating the multivariate b

R involves inverting a matrixthat does not have full rank when the parameters arerestricted to a simplex. The work-around is to onlylook at the first K � 1 values of these parameters.

The other commonly used test for pseudo-convergenceis to look at a trace plot for each parameter: a timeseries plot of the sampled parameter value against theMCMC cycle number. Multiple chains can be plot-ted on top of each other using di↵erent colors. If thechain is converged and the sample size is adequate, thetraceplot should look like white noise. If this is notthe case, a bigger MCMC sample is needed. Whileit is not particularly useful for an automated test ofpseudo-convergence, the traceplot is very useful for di-agnosing what is happening when the chains are notconverging. Some sample traceplots are given below.

Even if the chains have reached pseudo-convergence,

the size of the Monte Carlo error could still be an is-sue. Both rstan and coda compute an e↵ective samplesize for the Monte Carlo sample. This is the size ofa simple random sample from the posterior distribu-tion that would have the same Monte Carlo error asthe obtained dependent sample. This is di↵erent fordi↵erent parameters. If the e↵ective sample size is toosmall, then additional samples are needed.

Note that there are K(3I + 5) parameters whose con-vergence needs to be monitored. It is di�cult toachieve pseudo-convergence for all of these parame-ters, and exhausting to check them all. A reasonablecompromise seems to be to monitor only the 5K cross-student parameters, ↵, µ

0

, �0

, log(⌧0

) and �0

. JAGSmakes this process easier by allowing the analyst topick which parameters to monitor. Using JAGS, onlythe cross-student parameters can be monitored dur-ing the first MCMC run, and then a second shortersample of student-level parameters can be be obtainedthrough an additional run. The rstan package alwaysmonitors all parameters, including the ✓i and ⌘i pa-rameters which are not of particular interest.

When I and Ji are large, a run can take several hoursto several days. As the end of a run might occur dur-ing the middle of the night, it is useful to have a au-tomated test of convergence. The rule I have beenusing is to check to see if all b

R are less than a certainmaximum (by default 1.1) and if all e↵ective samplesizes are greater than a certain minimum (by default100) for all cross-student parameters. If these criteriaare not met, new chains of twice the length of the oldchain can be run. The traceplots are saved to a file forlater examination, as are some other statistics suchas the mean value of each chain. (These traceplotsand outputs are available at the web site mentionedabove.) It is generally not worthwhile to restart thechain for a third time. If pseudo-convergence has notbeen achieved after the second longer run, usually abetter solution is to reparameterize the model.

It is easier to extend the MCMC run using JAGS thanusing Stan. In JAGS, calling update again samplesadditional points from the Markov chain, starting atthe previous values, extending the chains. The currentversion of Stan (2.2.0)1 saves the compiled C++ code,but not the warm-up parameter values. So the sec-ond run is a new run, starting over from the warm-upphase. In this run, both the warm-up phase and thesampling phase should be extended, because a longer

1The Stan development team have stated that the abil-

ity to save and reuse the warm-up parameters are a high

priority for future version of Stan. Version 2.3 of Stan was

released between the time of first writing and publication

of the paper, but the release notes do not indicate that the

restart issue was addressed.

10

warm-up may produce a lower autocorrelation. In ei-ther case, the data from both runs can be pooled forinferences.

4.4 A simple test

To test the scripts and the viability of the model, Icreated some artificial data consistent with the modelin Section 2. To do this, I took one of the data setsuse by Li (2013) and ran the initial parameter algo-rithm described in Section 4.2 for K = 2, 3 and 4.I took the cross-student parameters from this exer-cise, and generated random parameters and data setsfor 10 students. This produced three data sets (forK = 2, 3 and 4) which are consistent with the modeland reasonably similar to the observed data. Allthree data sets and the sample parameters are avail-able on the web site, http://pluto.coe.fsu.edu/

mcmc-hierMM/.

For each data set, I fit three di↵erent hierarchical mix-ture models (K 0 = 2, 3 and 4, where K

0 is the valueused for estimation) using three di↵erent variationsof the algorithm: JAGS (RWM), Stan (HMC) withthe cross-student means constrained to be increasing,and Stan (HMC) with the means unconstrained. Forthe JAGS and Stan unconstrained run the chains weresorted (Section 4.1) on the basis of the cross-studentsmeans (µ

0

) before the convergence tests were run. Ineach case, the initial run of 3 chains with 5,000 ob-servations was followed by a second run of 3 chainswith 10,000 observations when the first one did notconverge. All of the results are tabulated at the website, including links to the complete output files andall traceplots.

Unsurprisingly, the higher the value of K 0 (number ofcomponents in the estimated model), the longer therun took. The runs for K 0 = 2 to between 10–15 min-utes in JAGS and from 30–90 minutes in Stan, whilethe runs for K 0 = 4 too just less than an hour in JAGSand between 2–3 hours in Stan. The time di↵erencebetween JAGS and Stan is due to the di↵erence inthe algorithms. HMC uses a more complex proposaldistribution than RWM, so naturally each cycle takeslonger. The goal of the HMC is to trade the longerrun times for a more e�cient (lower autocorrelation)sample, so that the total run time is minimized. Theconstrained version of Stan seemed to take longer thanthe unconstrained.

In all 27 runs, chain lengths of 15,000 cycles were notsu�cient to reach pseudo-convergence. The maximum(across all cross-student parameters) value for b

R variedbetween 1.3 and 2.2, depending on the run. It seemedto be slightly worse for cases in which K (number ofcomponents for data generation) was even and K

0 (es-

timated number of components) was odd or vice versa.The values of b

R for the constrained Stan model weresubstantially worse, basically confirming the findingsof Fruthwirth-Schnatter (2001).

The e↵ective sample sizes were much better for Stanand HMC. For the K

0 = 2 case, the smallest e↵ectivesample size ranged from 17–142, while for the uncon-strained Stan model it ranged from 805–3084. Thus,roughly 3 times the CPU time is producing a 5-folddecrease in the Monte Carlo error.

The following graphs compare the JAGS and Stan(Unconstrained model) outputs for four selected cross-student parameters. The output is from coda and pro-vides a traceplot on the left and a density plot for thecorresponding distribution on the right. Recall thatthe principle di↵erence between JAGS and Stan is theproposal distribution: JAGS uses a random walk pro-posal with a special distribution for the mixing param-eters and Stan uses the Hamiltonian proposal. Also,note that for Stan the complete run of 15,000 samplesis actually made up of a sort run of length 5,000 anda longer run of length 10,000; hence there is often adiscontinuity at iteration 5,000.

Although not strictly speaking a parameter, the de-viance (twice the negative log likelihood) can easilybe monitored in JAGS. Stan does not o↵er a deviancemonitor, but instead monitors the log posterior, whichis similar. Both give an impression of how well themodel fits the data. Figure 4 shows the monitors forthe deviance or log posterior. This traceplot is nearlyideal white noise, indicating good convergence for thisvalue. The value of b

R is less than the heuristic thresh-old of 1.1 for both chains, and the e↵ective sample sizeis about 1/6 of the 45,000 total Monte Carlo observa-tions.

Figure 5 shows the grand mean of the log pause timesfor the first component. The JAGS output (upperrow) shows a classic slow mixing pattern: the chainsare crossing but moving slowly across the support ofthe parameter space. Thus, for JAGS even thoughthe chains have nearly reached pseudo-convergence ( bRis just slightly greater than 1.1), the e↵ective samplesize is only 142 observations. A longer chain mightbe needed for a good estimate of this parameter. TheStan output (bottom row) looks much better, and thee↵ective sample size is a respectable 3,680.

The Markov chains for ↵

0,1 (the proportion of pausetimes in the first component) have not yet reachedpseudo convergence, but they are close (a longer runmight get them there). Note the black chain oftenranges above the values in the red and green chainsin the JAGS run (upper row). This is an indicationthat the chains may be stuck in di↵erent modes; the

11

Figure 4: Deviance (JAGS) and Log Posterior (Stan) plotsNote: To reduce the file size of this paper, this is a bitmap picture of the traceplot. The original pdf version isavailable at http://pluto.coe.fsu.edu/mcmc-hierMM/DeviancePlots.pdf.

12

Figure 5: Traceplots and Density plots for µ0,1

Note: To reduce the file size of this paper, this is a bitmap picture of the traceplot. The original pdf version isavailable at http://pluto.coe.fsu.edu/mcmc-hierMM/mu0Plots.pdf.

13

Figure 6: Traceplots and Density plots for ↵0,1

Note: To reduce the file size of this paper, this is a bitmap picture of the traceplot. The original pdf version isavailable at http://pluto.coe.fsu.edu/mcmc-hierMM/alpha0Plots.pdf.

14

shoulder on the density plot also points towards mul-tiple modes. The plot for Stan looks even worse, afteriteration 5,000 (i.e., after the chain was restarted), thered chain has moved into the lower end of the supportof the distribution. This gives a large shoulder in thecorresponding density plot.

The chains for the variance of the log precisions for thefirst component, �

0,1 (Figure 7), are far from pseudo-

convergence with bR = 1.73. In the JAGS chains (top

row) the black chain seems to prefer much smaller val-ues for this parameter. In the Stan output, the greenchain seems to be restricted to the smaller values inboth the first and second run. However, between thefirst and second run the black and red chains swapplaces. Clearly this is the weakest spot in the model,and perhaps a reparameterization here would help.

Looking at Figures 6 and 7 together, a patternemerges. There seems to be (at least) two distinctmodes. One mode (black chain JAGS, green chain inStan) has higher value for ↵

0,1 and a lower value for�

0,1, and the other one has the reverse pattern. Thisis an advantage of the MCMC approach over an EMapproach. In particular, if the EM algorithm was onlyrun once from a single starting point, the existence ofthe other mode might never be discovered.

4.5 Data Point Labeling

For some applications, the analysis can stop after es-timating the parameters for the students units, ⇡i, µi

and ⌧ i (or �i as desired). In other applications it isinteresting to assign the individual pauses to compo-nents, that is, to assign values to Zi,j .

When ⇡i, µi and �i are known, it is simple to calculate

pi,j,k = Pr(Zi,j = k). Let ⇡(c,r)i , µ(c,r)

i and �(c,r)i be

the draws of the Level 1 parameters from Cycle r ofChain c (after the data have been sorted to identifythe components). Now define,

q

(c,r)i,j,k = ⇡

(c,r)i,k �(

Yi,j � µ

(c,r)i,k

�

(c,r)i,k

), (17)

and set Q(c,r)i,j =

PKk=1

q

(c,r)i,j,k . The distribution for Zi,j

for MCMC draw (c, r) is then p

(c,r)i,j,k = q

(c,r)i,j,k /Q

(c,r)i,j .

The MCMC estimate for pi,j,k is just the aver-

age over all the MCMC draws of p

(c,r)i,j,k , that is

1

CR

PCc=1

PRr=1

p

(c,r)i,j,k .

If desired, Zi,j can be estimated as the maximum overk of pi,j,k, but for most purposes simply looking at theprobability distribution pi,j provides an adequate, ifnot better summary of the available information aboutthe component from which Yi,j was drawn.

4.6 Additional Level 2 Units (students)

In the pause time application, the goal is to be ableto estimate fairly quickly the student-level parametersfor a new student working on the same essay. As theMCMC run is time consuming, a fast method for an-alyzing data from new student would be useful.

One approach would be to simply treat Equations 2,3 and 4 as prior distributions, plugging in the pointestimates for the cross-student parameters as hyper-parameters, and find the posterior mode of the ob-servations from the new data. There are two problemswith this approach. First, the this method ignores anyremaining uncertainty about the cross-student param-eters. Second, the existing mixture fitting softwarepackages (FlexMix and mixtools) do not provide anyway to input the prior information.

A better approach is to do an additional MCMC run,using the posterior distributions for the cross-studentparameters from the previous run as priors from theprevious run (Mislevy et al., 1999). If the hyperpa-rameters for the cross-student priors are passed tothe MCMC sampler as data, then the same JAGS orStan model can be reused for additional student pauserecords. Note that in addition to the MCMC codeStan provides a function that will find the maximumof the posterior. Even if MCMC is used at this step,the number of Level 2 units, I, will be smaller and thechains will reach pseudo-convergence more quickly.

If the number of students, I, is large in the originalsample (as is true in the original data set), then asample of students can be used in the initial modelcalibration, and student-level parameters for the oth-ers can be estimated later using this method. In par-ticular, there seems to be a paradox estimating mod-els using large data sets with MCMC. The larger thedata, the slower the run. This is not just becauseeach data point needs to be visited within each cycleof each Markov chain. It appears, especially with hi-erarchical models, that more Level 2 units cause thechain to have higher autocorrelation, meaning largerruns are needed to achieve acceptable Monte Carlo er-ror. For the student keystroke logs, the “initial data”used for calibration was a sample of 100 essays fromthe complete collection. There is a obvious trade-o↵between working with smaller data sets and being ableto estimate rare events: I = 100 seemed like a goodcompromise.

5 Model Selection

A big question in hierarchical mixture modeling is“What is the optimal number of components, K?”Although there is a way to add a prior distribution

15

Figure 7: Traceplots and Density plots for �0,1

Note: To reduce the file size of this paper, this is a bitmap picture of the traceplot. The original pdf version isavailable at http://pluto.coe.fsu.edu/mcmc-hierMM/gamma0Plots.pdf.

16

over K and learn this as part of the MCMC pro-cess, it is quite tricky and the number of parametersvaries for di↵erent values ofK. (Geyer, 2011, describesthe Metropolis-Hastings-Green algorithm which is re-quired for variable dimension parameter spaces). Inpractice, analysts often believe that the optimal valueof K is small (less than 5), so the easiest way to de-termine a value of K is to fit separate models forK = 2, 3, 4, . . . and compare the fit of the models.

Note that increasing K should always improve the fitof the model, even if the extra component is just usedto trivially fit one additional data point. Consequently,analysts use penalized measures of model fit, such asAIC to chose among the models.

Chapter 7 of Gelman et al. (2013) gives a goodoverview of the various information criteria, so thecurrent section will provide only brief definitions. Re-call Equation 5 that described the log likelihood ofthe complete data Y as a function of the completeparameter ⌦. By convention, the deviance is twicethe negative log likelihood: D(⌦) = �2L(Y|⌦). Thelower the deviance, the better the fit of the data to themodel, evaluated at that parameter. As students areindependent given the cross-student parameters, andpause times are independent given the student-levelparameters, D(e⌦), for some value of the parameterse⌦ can be written as a sum:

D(e⌦) =IX

i=1

JiX

j=1

logQi,j(e⌦) , (18)

where Qi,j(e⌦) is the likelihood of data point Yi,j , thatis:

Q

i,j(e⌦) =kX

k=1

g⇡i,k�(Yi,j � gµi,k

g�i,k). (19)

Note that the cross-student parameters drop out ofthese calculations.

The Akaike information criterion (AIC) takes the de-viance as a measure of model fit and penalizes it forthe number of parameters in the model. Let b⌦ be themaximum likelihood estimate of the parameter. ThenAIC is defined as:

AIC = D(b⌦) + 2 ⇤m (20)

where m = K(2IK + 4) + (K � 1)(I + 1) + 1 is thenumber of free parameters in the model. Note thatthe MCMC run does not provide a maximum likeli-hood estimate. A pseudo-AIC can be computed bysubstituting ⌦, the posterior mean of the parameters⌦. Note that ⌦ must be calculated after the parame-ters in each MCMC cycle are permuted to identify thecomponents.

Gelman et al. (2013) advocate using a new measure ofmodel selection called WAIC. WAIC makes two sub-stitutions in the definition of AIC above. First, it usesD(⌦) in place of D(⌦). Second, it uses a new way ofcalculating the dimensionality of the model. There aretwo possibilities:

pWAIC1= 2

IX

i=1

JiX

j=1

(logE[Qi,j(⌦)]� E[logQi, j(⌦)]) ,

(21)

pWAIC2= 2

IX

i=1

JiX

j=1

Var(Qi,j(⌦)) . (22)

The expectation and variance are theoretically definedover the posterior distribution and are approximatedusing the MCMC sample. The final expression forWAIC is:

WAIC = D(⌦) + 2 ⇤mWAIC . (23)

For all of the model fit statistics discussed here: AIC,WAIC

1

and WAIC2

, the model with the lowest valueis the best. One way to chose the best value of K is tocalculate all of these statistics for multiple values of Kand choose the value of K which has the lowest valuefor all 5 statistics. Hopefully, all of the statistics willpoint at the same model. If there is not true, that isoften evidence that the fit of two di↵erent models istoo close to call.

In the 27 runs using the data from known values of K,simply using the lowest WAIC value was not su�cientto recover the model. As the Markov chains have notyet reached pseudo-convergence, the WAIC values maynot be correct, but there was fairly close agreement onboth WAIC

1

and WAIC2

for both the JAGS and Stan(unconstrained) models. However, the values of bothWAIC

1

and WAIC2

were also fairly close for all valuesofK 0 (number of estimated components). For examplewhen K = 2 the WAIC

1

value was 1132, 1132 and1135 for K 0 = 2, 3 and 4 respectively when estimatedwith JAGS and 1132, 1131, and 1132 when estimatedwith Stan. The results for WAIC

2

were similar. Itappears as if the common practice of just picking thebest value of WAIC to determine K is not su�cient.In particular, it may be possible to approximate thedata quite well with a model which has an incorrectvalue of K.

6 Conclusions

The procedure described above is realized as code inR, Stan and JAGS and available for free at http://

pluto.coe.fsu.edu/mcmc-hierMM/. Also available,are the simulated data for trying out the models.

17

These scripts contain the functions necessary to auto-matically set starting parameters for these problems,identify the components, and to calculate WAIC modelfit statistics.

The scripts seem to work fairly well for a small numberof students units (5 I 10). With this sample size,although JAGS has the faster run time, Stan has alarger e↵ective sample size. Hamiltonian Monte Carlo.With large sample sizes I = 100, Ji ⇡ 100 even theHMC has high autocorrelation and both JAGS andStan are extremely slow. Here the ability of JAGS toallow the user to selectively monitor parameters andto extend the chains without starting over allows forbetter memory management.

The code supplied at the web site solves two key tech-nical problems: the post-run sorting of the mixturecomponents (Fruthwirth-Schnatter, 2001) and the cal-culation of the WAIC model-fit indexes. Hopefully,this code will be useful for somebody working on asimilar application.

The model proposed in this paper has two problems,even with data simulated according to the model. Thefirst is the slow mixing. This appears to be a prob-lem with multiple modes, and indicates some possiblenon-identifiability in the model specifications. Thus,further work is needed on the form of the model. Thesecond problem is that the WAIC test does not ap-pear to be sensitive enough to recover the number ofcomponents in the model. As the original purpose ofmoving to the hierarchical mixture model was to dis-cover if there were rare components that could not bedetected in the single-level hierarchical model, the cur-rent model does not appear to be a good approach forthat purpose.

In particular, returning to the original application offitting mixture models to student pauses while writing,the hierarchical part of the model seems to be creat-ing as many problems as it solves. It is not clearlybetter than fitting the non-hierarchical mixture modelindividually to each student. Another problem it hasfor this application is the assumption that each pauseis independent. This does not correspond with whatis known about the writing process: typically writ-ers spend long bursts of activities simply doing tran-scription (i.e., typing) and only rarely pause for higherlevel processing (e.g., spelling, grammar, word choice,planning, etc). In particular, rather than getting thismodel to work for this application, it may be better tolook at hidden Markov models for individual studentwritings.2

The slow mixing of the hierarchical mixture model

2Gary Feng and Paul Deane, private communication.

for large data sets indicates that there may be addi-tional transformation of this model that are necessaryto make it work properly. Hopefully, the publication ofthe source code will allow other to explore this modelmore fully.

References

Almond, R. G., Deane, P., Quinlan, T., Wagner,M., & Sydorenko, T. (2012). A prelimi-nary analysis of keystroke log data from atimed writing task (Research Report No.RR-12-23). Educational Testing Service. Re-trieved from http://www.ets.org/research/

policy research reports/publications/

report/2012/jgdg

Benaglia, T., Chauveau, D., Hunter, D. R., & Young,D. (2009). mixtools: An R package for ana-lyzing finite mixture models. Journal of Sta-tistical Software, 32 (6), 1–29. Retrieved fromhttp://www.jstatsoft.org/v32/i06/

Brooks, S., Gelman, A., Jones, G. L., & Meng, X.(Eds.). (2011). Handbook of markov chain montecarlo. CRC Press.

Deane, P. (2012). Rethinking K-12 writing assessment.In N. Elliot & L. Perelman (Eds.), Writing as-sessment in the 21st century. essays in honor ofEdward M. white (pp. 87–100). Hampton Press.

Fruthwirth-Schnatter, S. (2001). Markov chainMonte Carlo estimation of classical and dynamicswitching and mixture models. Journal of theAmerican Statistical Association, 96 (453), 194–209.

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B.,Vehtari, A., & Rubin, D. B. (2013). Bayesiandata analysis (3rd ed.). Chapman and Hall. (Thethird edition is revised and expanded and hasmaterial that the earlier editions lack.)

Gelman, A., & Shirley, K. (2011). Inference fromsimulations and monitoring convergence. InS. Brooks, A. Gelman, G. L. Jones, & X. Meng(Eds.), Handbook of markov chain monte carlo(pp. 163–174). CRC Press.

Geyer, C. J. (2011). Introduction to Markovchain Monte Carlo. In S. Brooks, A. Gelman,G. L. Jones, & X. Meng (Eds.), Handbook ofmarkov chain monte carlo (pp. 3–48). CRCPress.

Gruen, B., & Leisch, F. (2008). FlexMix version2: Finite mixtures with concomitant variablesand varying and constant parameters. Journalof Statistical Software, 28 (4), 1–35. Retrievedfrom http://www.jstatsoft.org/v28/i04/

Li, T. (2013). A Bayesian hierarchical mixture ap-proach to model timing data with application to

18

writing assessment (Unpublished master’s the-sis). Florida State University.

McLachlan, G., & Krishnan, T. (2008). The EM al-gorithm and extensions (2nd ed.). Wiley.

McLachlan, G., & Peel, D. (2000). Finite mixturemodels. Wiley.

Mislevy, R. J., Almond, R. G., Yan, D., & Stein-berg, L. S. (1999). Bayes nets in educationalassessment: Where the numbers come from. InK. B. Laskey & H. Prade (Eds.), Uncertainty inartificial intelligence ’99 (p. 437-446). Morgan-Kaufmann.

Neal, R. M. (2011). MCMC using Hamiltonian dynam-ics. In MCMCHandbook (Ed.), (pp. 113–162).

Plummer, M. (2012, May). JAGS version 3.2.0user manual (3.2.0 ed.) [Computer softwaremanual]. Retrieved from http://mcmc-jags

.sourceforge.net/

Plummer, M., Best, N., Cowles, K., & Vines, K.(2006). Coda: Convergence diagnosis and out-put analysis for mcmc. R News, 6 (1), 7–11.Retrieved from http://CRAN.R-project.org/

doc/Rnews/

R Core Team. (2014). R: A language and environ-ment for statistical computing [Computer soft-ware manual]. Vienna, Austria. Retrieved fromhttp://www.R-project.org/

Stan Development Team. (2013). Stan: A C++ libraryfor probability and sampling (Version 2.2.0 ed.)[Computer software manual]. Retrieved fromhttp://mc-stan.org/

Thomas, A., O’Hara, B., Ligges, U., & Sturtz, S.(2006). Making BUGS open. R News, 6 , 12-17.Retrieved from http://www.rni.helsinki.fi/

~boh/publications/Rnews2006-1.pdf

Thomas, A., Spiegelhalter, D. J., & Gilks, W. R.(1992). BUGS: A program to performBayesian inference using Gibbs sampling. InJ. M. Bernardo, J. O. Berger, A. P. Dawid, &A. F. M. Smith (Eds.), Bayesian statistics 4. (pp.837–842). Clarendon Press.

Acknowledgments

Tingxuan Li wrote the first version of much of theR code as part of her Master’s thesis. Paul Deaneand Gary Feng of Educational Testing Service havebeen collaborators on the essay keystroke log analysiswork which is the inspiration for the present research.Various members of the JAGS and Stan users groupsand development teams have been helpful in answer-ing emails and postings related to these models, andKathy Laskey and several anonymous reviewers havegiven me useful feedback (to which I should have paidcloser attention) on the draft of this paper.

19