using julia and jumpfor personalized product ...web.mit.edu/.../ellipsoidconjoint_uai17.pdf ·...

UsingJuliaandJuMP forpersonalizedproductrecommendations:Ellipsoidalmethodsforadaptivechoice-basedconjointanalysis

JuanPabloVielmaMassachusettsInstituteofTechnology

JointworkwithDenisSaure

UniversidadAdolfoIbañez,Santiago,Chile.August,2017.

3/9/16, 12:08 PMAcademic Page of Juan Pablo Vielma

Page 3 of 3http://www.mit.edu/~jvielma/

Jennifer ChallisE62-571, 100 Main Street,Cambridge, MA 02142(617) 324-4378jchallis at mit dot edu

CollaboratorsShabbir Ahmed, Daniel Bienstock, Daniel Dadush, Sanjeeb Dash, Santanu S. Dey, Iain Dunning, RodolfoCarvajal, Luis A. Cisternas, Miguel Constantino, Daniel Espinoza, Alexandre S. Freire, Marcos Goycoolea,Oktay Günlük, Joey Huchette, Nathalie E. Jamett, Ahmet B. Keha, Mustafa R. Kılınç, Guido Lagos, MilesLubin, Sajad Modaresi, Sina Modaresi, Eduardo Moreno, Diego Morán, Alan T. Murray, George L.Nemhauser, Luis Rademacher, David M. Ryan, Denis Saure, Alejandro Toriello, Andres Weintraub, SercanYıldız, Tauhid Zaman

Affiliations

Links

Back to top© 2013 Juan Pablo Vielma | Last updated: 02/29/2016 00:47:25 | Based on a template design by AndreasViklund





Affiliations

Links


&

HopeIamPreachingtotheChoir!

JuliaandJuMP TutorialatUniversidadAdolfoIbáñez,Santiago,Chile.January,2014.More variables!

sandwich = [:italiano,:chacarero]

@defVar(m, 0 <= P[sandwich] <= 1)

@defVar(m, i <= x[i=1:5] <= 2i)

getValue(x[3])

getDual(P[:italiano]) # reduced cost

2 /37

21stCenturyProgramming/ModellingLanguages

• Open-sourceandfree!• DevelopedatMIT• “Floatslikepython/matlab,

stingslikeC/Fortran”• Easytouseandwide

libraryecosystem(specializedandfrontend)

• OnlylanguagebesidesC/C++/Fortrantoscaleto1Petaflop!





Affiliations

Links






Affiliations

Links

Back to top© 2013 Juan Pablo Vielma | Last updated: 02/29/2016 00:47:25 | Based on a template design by AndreasViklund• Open-sourceandfree!

• Developedat

• Optimizationmodellinglanguageandinterphase

• Easytouseandadvanced• IntegratedintoJulia

8/16/17, 7(33 PM

Page 1 of 1http://www.mit.edu/~jvielma/orc.svg

8/16/17, 7(32 PM

Page 1 of 1http://www.mit.edu/~jvielma/sloan-new.svg

Createdbystudents





Affiliations

Links


IainDunning,MilesLubinandJoeyHuchette

Advisor/Boss/ …

JuanPabloVielma

8/16/17, 7(32 PM


CommunityDevelopers!"#$%&'

SoftwareEngineerJarrettRevels

8/16/17, 7(32 PM


8/16/17, 7(33 PM

Page 1 of 1http://www.mit.edu/~jvielma/orc.svg

Godfather? JuMP-Suit?

ProductRecommendation,Choice-BasedConjointAnalysis,andExperimentalDesign

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99

Weight 15.68lb 7.5lb

Prefer � �

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99

Weight 15.68ounces 7.5ounces

Prefer � �

Feature TG-4 G9

Waterproof Yes No

Prize $249.99 $399.99

Weight 7.36lb 7.5lb

Prefer � �

Feature TG-4 Galaxy2

Waterproof Yes No

Prize $249.99 $399.99

Viewfinder Electronic Optical

Prefer � �

Werecommend:

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

Motivation:(Custom)ProductRecommendations

Towards CBCA-BasedRecommendations

• Individualpreferenceestimateswithfewquestions

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

Feature TG-4 G9

Waterproof Yes No

Prize $249.99 $399.99

Weight 7.36lb 7.5lb

Prefer � �


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

Werecommend:

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

• AdaptiveQuestions:– Fastquestionselection– Picknext questiontoreduceuncertainty

– Quantifyestimatevariance

• Favorablepropertiesforfuture:– Intuitivegeometricmodel(e.g.RobustOpt.)

– Parametricmodel

Choice-basedConjointAnalysis

Feature Chewbacca BB-8

Wookiee Yes No

Droid No Yes

Blaster Yes No

Iwould buytoy ☐ ☐

x

2x

1Product Profile

0

@010

1

A = x

2

ParametricModel=LogisticRegression

• MNLRandomLinear Utilities(dproductfeatures)

• Utilitymaximizingcustomer:

• Noise=responseerror:

• Regression:

x

1 ⌫ x

2 , U1“�”U2

noise (gumbel)part-worths (weights)product profile

Up = � · xp + ✏p =Xd

i=1�ix

pi + ✏p p 2 {1, 2}

x

1 ⌫ x

2 , � · z “�” 0 , sign(� · z) = 1

z := x

1 � x

2, y := sign(� · z) 2 {0, 1}

P�x

1 ⌫ x

2 | ��=

1

1 + e

��·(x1�x

2)

BayesianUpdateAfteraQuestionisAnswered

Prior distribution

� ⇠ N(µ,⌃)

Answer likelihood

Posterior distribution

L (y | �, z) g (� | y, z )

g (� | y, z ) / � (� ; µ,⌃)L (y | �, z)

L (y | �, z) =�1 + e�y�·z��1

y = sign(� · z)

PickNextQuestionToReducePosterior“Variance”

-4 -2 0 2 4

0.05

0.10

0.15

0.20

0.25

Prior Distribution

of �

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

BayesianUpdate

-4 -2 0 2 4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

PosteriorDistribution

MCMC


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

-4 -2 0 2 4

0.05

0.10

0.15

0.20

0.25

0.30

PosteriorDistribution

BayesianUpdate

• Multivariateversionwithuncertaintyinanswer:D-Error:

� ⇠ N(µ,⌃)

f(z, µ,⌃) := Ey,�

n

(det cov(� | y, z))1/do

BayesianD-OptimalQuestionSelection

🙂 State-of-the-artMCMCtoolseasytoaccessfrom Julia(Stan.jl):

🙁Minutestoevaluateonequestionfor12features=Months tofindbest!

😒 “FisherInformationapproximations”:• Minutes tofind“best”byenumeration• Highdimensionalnon-convexoptimization.• Low/Highvariancenumericalissues.

8/11/17, 6(23 PMStan - Stan

Page 1 of 2http://mc-stan.org/

Registration is now open for StanCon 2018!

Stan is a state-of-the-art platform for statistical modeling and high-performancestatistical computation. Thousands of users rely on Stan for statistical modeling, dataanalysis, and prediction in the social, biological, and physical sciences, engineering, andbusiness.

Users specify log density functions in Stan’s probabilistic programming language and get:

full Bayesian statistical inference with MCMC sampling (NUTS, HMC)

approximate Bayesian inference with variational inference (ADVI)

penalized maximum likelihood estimation with optimization (L-BFGS)

Stan’s math library provides differentiable probability functions & linear algebra (C++

! Menu

Stan

®

8/11/17, 6(23 PMStan - Stan

Page 1 of 2http://mc-stan.org/

Registration is now open for StanCon 2018!

Stan is a state-of-the-art platform for statistical modeling and high-performancestatistical computation. Thousands of users rely on Stan for statistical modeling, dataanalysis, and prediction in the social, biological, and physical sciences, engineering, andbusiness.

Users specify log density functions in Stan’s probabilistic programming language and get:

full Bayesian statistical inference with MCMC sampling (NUTS, HMC)

approximate Bayesian inference with variational inference (ADVI)

penalized maximum likelihood estimation with optimization (L-BFGS)

Stan’s math library provides differentiable probability functions & linear algebra (C++

! Menu

Stan

®

Alternative:FastGeometric/OptimizationModels

�1

�2

�1

�2

�1

�2

Answer hyperplanePosterior set

Prior set

� 2 Pi � · z � 0 � 2 Pi+1

Method Response ErrorPolyhedralMethod(Toubiaetal.‘03,’04) NoProbabilisticPolyhedralMethod(T.etal.‘07) Yes,≈Bayesian

RobustMethod(BertsimasandO’Hair‘13) Yes,Robust

Bayesianv/sGeometric

Bayesian Geometric

ResponseError MNL None /Non-MNL

Update IntegrationorMCMC

SimpleLinearAlgebra

QuestionSelection

Integration+Enumeration MIP





Affiliations

Links






Affiliations

Links


• EllipsoidalMethod:–MIP, andbridgestheGAP(QuestionSelection)

MixedIntegerProgrammingandOptimalQuestionSelection

NextQuestion:Reduce“Variance”/D-Error

-4 -2 0 2 4

0.05

0.10

0.15

0.20

0.25

Prior Distribution

of �

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

-4 -2 0 2 4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Posterior Distribution

Update


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

-4 -2 0 2 4

0.05

0.10

0.15

0.20

0.25

0.30

Posterior Distribution

� ⇠ N(µ,⌃)

minx

1 6=x

22{0,1}df

�x

1 � x

2, µ,⌃

�

f(z, µ,⌃) := Ey,�

n

(det cov(� | y, z))1/do

LowDimensionalReformulationofD-Error

• D-efficiency=Non-convexfunction off(d, v)

Canevaluatewith1-dimintegral🙂

f(d, v)

LinearMIPformulation(standardlinearization)

PiecewiseLinear(PWL)Interpolation

mean: d := µ · z

variance: v := z0 ·⌃ · z

f(z)

AlignswithselectioncriteriafromToubiaetal.’04:minimizemeanandmaximizevariance

�f(d, v)

min f(d, v)

s.t.

µ ·�x

1 � x

2�= d

�x

1 � x

2�0 ·⌃ ·

�x

1 � x

2�= v

A

1x

1 +A

2x

2 b

x

1 6= x

2

x

1, x

2 2 {0, 1}n

LinearMIP:LinearizeQuad+PWLFormulation

linearize x

ki · xl

j

EasytoBuildthrough&

• PiecewiseLinearOpt.jl (Huchette andV.2017)

5/27/17, 5:15 PMjoehuchette/PiecewiseLinearOpt.jl: Optimizing over piecewise linear functions

Page 2 of 2https://github.com/joehuchette/PiecewiseLinearOpt.jl

m=Model()

@variable(m,x)

To model the graph of a piecewise linear function f(x), take d as some set of breakpoints along the real line, and fd=[f(x)forxind] as the corresponding function values. You can model this function in JuMP using the following function:

z=piecewiselinear(m,x,d,fd)

@objective(m,Min,z)#minimizef(x)

For another example, think of a piecewise linear approximation for for the function $f(x,y) = exp(x+y)$:

usingJuMP,PiecewiseLinearOpt

m=Model()

@variable(m,x)

@variable(m,y)

z=piecewiselinear(m,x,y,0:0.1:1,0:0.1:1,(u,v)->exp(u+v))

@objective(m,Min,z)

Current support is limited to modeling the graph of a continuous piecewise linear function, either univariate or bivariate,with the goal of adding support for the epigraphs of lower semicontinuous piecewise linear functions.

Contact GitHub API Training Shop Blog About© 2017 GitHub, Inc. Terms Privacy Security Status Help

min exp(x+ y)

s.t.

x, y 2 [0, 1]

10

10

Automaticallyselect∆

Automaticallyconstructformulation(easilychosen)





Affiliations

Links






Affiliations

Links


EllipsoidalMethod:PuttingEverythingTogether

MIP-basedAdaptiveQuestionnaires

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

Feature TG-4 G9

Waterproof Yes No

Prize $249.99 $399.99

Weight 7.36lb 7.5lb

Prefer � �


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

Werecommend:

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �

• Optimalone-steplook-aheadmoment-matchingapproximateBayesianapproach=EllipsoidalMethod

E�� |Y,X1, X2

�

cov

�� |Y,X1, X2

�

OptimalOne-StepLook-Ahead=MIP

Prior distribution

Feature SX530 RX100

Zoom 50x 3.6x

Prize $249.99 $399.99


Prefer � �


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

Feature TG-4 G9

Waterproof Yes No

Prize $249.99 $399.99

Weight 7.36lb 7.5lb

Prefer � �

minx

1,x

2f

�x

1, x

2�

• SolvewithMIPformulation

• 1-dimnumericalintegration:QuadGK.jl

� ⇠ N�µi,⌃i

�

minx

1,x

2,d,v2Q

f(d, v)

Moment-MatchingApproximateBayesianUpdate

Prior distribution


Waterproof Yes No

Prize $249.99 $399.99


Prefer � �

Answer likelihood

Posterior distribution

� ⇠ N�µi,⌃i

��

approx.⇠ N�µi+1,⌃i+1

�

• µ

i+1= E

�� | y, x1

, x

2�

• ⌃

i+1= cov

�� | y, x1

, x

2� • 1-dintegral: I(d, v)

SimulationExperiments

• 16questions,2options,12features• SimulateMNLresponseswithknown𝛽*

• 100individual𝛽* sampledfromN(𝜇,𝛴)prior• Methods:– Polyhedral,Prob.Polyhedral,RobustandEllipsoidal– Allgetsameellipsoidalprior– All<30’’inter-question(exceptrobust<90’’)

• Metrics:– RMSEof𝛽 estimator,errorinmarketshareandD-eff.– Normalizedvalues=smallerbetter– Versions:MethodandBayesianEstimator– Sensitivity:Wrongprior𝜇,allerrorsinfirst/secondhalf

D-EfficiencyforIndividualBayesian

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE

▲ Probabilistic Polyhedral ■ Robust ● Bayesian Ellipsoid ◇ Polyhedral

Baseline

WrongPrior SecondHalfErrors

FirstHalfErrors▲▲▲▲▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■■■ ■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ●

◇◇◇◇◇◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeff2▲

▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇

◇

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeff2

▲

▲▲

▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeffdim

▲

▲ ▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■■■■ ■

■■ ■ ■ ■ ■ ■ ■ ■ ■

● ●

●

●

●●

●● ● ● ● ● ● ● ● ● ●

◇

◇

◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeffdim


▲▲

▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■■ ■

■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

●●

●● ● ● ● ● ● ● ● ● ● ●

◇◇

◇◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeff2▲

▲

▲

▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇

◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeff2

▲

▲

▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■

■ ■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeffdim

▲▲

▲

▲▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■ ■ ■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

● ●

●● ● ● ● ● ● ● ● ● ● ● ● ●

◇◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.5

1.5

HBDeffdim


▲▲▲▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■■ ■

■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ●

◇◇◇

◇◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeff2▲

▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■ ■

■ ■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

● ● ● ● ● ● ● ● ●

●

● ● ● ● ●

◇

◇ ◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeff2

▲

▲

▲

▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■

■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeffdim▲

▲▲ ▲

▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■■ ■

■■■

■

■ ■ ■■■ ■ ■ ■ ■

●●

●●

●

● ● ● ● ● ●

●

● ● ● ● ●

◇

◇ ◇◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeffdim


▲▲▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ●

◇◇◇◇◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeff2▲

▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■

■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

HBDeff2

▲

▲

▲

▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■

■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●

●

●●

● ● ● ● ● ● ● ● ● ● ● ● ●

◇

◇

◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.2

0.4

0.6

0.8

BDeffdim

▲

▲

▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■ ■

■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

● ● ●

●

●

● ●●

●● ● ● ● ● ● ● ●

◇ ◇◇ ◇

◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.0

0.5

1.5

HBDeffdim


RMSEforMethodsEstimator

▲

▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■

● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●

◇

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●●

●●

●●

●●

●●

●● ●

● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.70

0.75

0.80

0.85

0.90

0.95

BRMSE

▲▲ ▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■ ■

■ ■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ●

●● ●

● ●●

● ● ●

● ● ● ● ●

◇◇

◇◇

◇ ◇ ◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.90

0.95

1.00

1.05

1.10HBRMSE

▲

▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■

● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●

◇

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE


▲▲ ▲ ▲

▲ ▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■ ■■ ■ ■

■■ ■ ■ ■

■ ■ ■ ■■

●●

●●

● ●●

●● ● ● ●

●● ● ● ●

◇◇

◇

◇◇ ◇

◇◇

◇◇

◇◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.1

1.2

1.3

1.4

RMSE

▲▲

▲

▲▲

▲

▲▲

▲▲

▲ ▲▲

▲ ▲▲

▲■■

■■

■■

■■

■■

■■

■■

■■ ■

●●

●

●●

●

●●

●●

●●

●●

●●

●

◇◇

◇

◇◇

◇

◇◇

◇◇

◇◇

◇ ◇◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.95

1.00

1.05

1.10

BRMSE

▲ ▲

▲▲

▲

▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ●◇ ◇

◇◇

◇◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.2

1.4

1.6

HBRMSE

▲▲ ▲ ▲

▲ ▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■ ■■ ■ ■

■■ ■ ■ ■

■ ■ ■ ■■

●●

●●

● ●●

●● ● ● ●

●● ● ● ●

◇◇

◇

◇◇ ◇

◇◇

◇◇

◇◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.1

1.2

1.3

1.4

RMSE


▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲

▲▲

▲ ▲ ▲■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■

■ ■

● ● ● ● ● ● ● ● ●●

●

●●

●● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

◇ ◇◇

◇◇

◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE▲

▲ ▲▲

▲▲ ▲ ▲

▲▲

▲

▲▲

▲▲ ▲ ▲

■■ ■ ■

■■

■ ■■ ■

■■

■■

■ ■ ■

●●

●●

●●

●●

●●

●

●

●

●

●●

●

◇◇ ◇

◇ ◇◇

◇ ◇ ◇◇

◇

◇◇

◇◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.70

0.75

0.80

0.85

0.90

0.95

BRMSE

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■●

●

● ● ● ● ● ● ● ● ●

●

●●

●● ●◇ ◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.2

1.4

1.6

1.8

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲

▲▲

▲ ▲ ▲■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■

■ ■

● ● ● ● ● ● ● ● ●●

●

●●

●● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

◇ ◇◇

◇◇

◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE


▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE


▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE


Baseline


FirstHalfErrors

RMSEforIndividualBayesianEstimator

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE


Baseline


FirstHalfErrors

▲

▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■

● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●

◇

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●●

●●

●●

●●

●●

●● ●

● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.70

0.75

0.80

0.85

0.90

0.95

BRMSE

▲▲ ▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■■ ■

■ ■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ●

●● ●

● ●●

● ● ●

● ● ● ● ●

◇◇

◇◇

◇ ◇ ◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.90

0.95

1.00

1.05

1.10HBRMSE

▲

▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■

● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●

◇

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE


▲▲ ▲ ▲

▲ ▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■ ■■ ■ ■

■■ ■ ■ ■

■ ■ ■ ■■

●●

●●

● ●●

●● ● ● ●

●● ● ● ●

◇◇

◇

◇◇ ◇

◇◇

◇◇

◇◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.1

1.2

1.3

1.4

RMSE

▲▲

▲

▲▲

▲

▲▲

▲▲

▲ ▲▲

▲ ▲▲

▲■■

■■

■■

■■

■■

■■

■■

■■ ■

●●

●

●●

●

●●

●●

●●

●●

●●

●

◇◇

◇

◇◇

◇

◇◇

◇◇

◇◇

◇ ◇◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.95

1.00

1.05

1.10

BRMSE

▲ ▲

▲▲

▲

▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ●◇ ◇

◇◇

◇◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.2

1.4

1.6

HBRMSE

▲▲ ▲ ▲

▲ ▲▲ ▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■ ■■ ■ ■

■■ ■ ■ ■

■ ■ ■ ■■

●●

●●

● ●●

●● ● ● ●

●● ● ● ●

◇◇

◇

◇◇ ◇

◇◇

◇◇

◇◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.1

1.2

1.3

1.4

RMSE


▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲

▲▲

▲ ▲ ▲■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■

■ ■

● ● ● ● ● ● ● ● ●●

●

●●

●● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

◇ ◇◇

◇◇

◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE▲

▲ ▲▲

▲▲ ▲ ▲

▲▲

▲

▲▲

▲▲ ▲ ▲

■■ ■ ■

■■

■ ■■ ■

■■

■■

■ ■ ■

●●

●●

●●

●●

●●

●

●

●

●

●●

●

◇◇ ◇

◇ ◇◇

◇ ◇ ◇◇

◇

◇◇

◇◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.70

0.75

0.80

0.85

0.90

0.95

BRMSE

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■●

●

● ● ● ● ● ● ● ● ●

●

●●

●● ●◇ ◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.2

1.4

1.6

1.8

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲

▲▲

▲ ▲ ▲■ ■

■■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■■

■ ■

● ● ● ● ● ● ● ● ●●

●

●●

●● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

◇ ◇◇

◇◇

◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.7

0.8

0.9

1.0

1.1

1.2

1.3

RMSE


▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE


MarketshareforBaseline

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

●● ● ●

●●

●●

●●

●●

● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

BRMSE

▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■

■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■● ● ●

● ●● ● ● ● ● ● ● ● ● ● ● ●

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1.0

1.5

2.0

2.5

3.0

3.5

HBRMSE

▲▲

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

■ ■ ■■

■ ■ ■■ ■ ■ ■

■ ■ ■■

■ ■●

● ● ● ● ●●

●●

●●

● ● ● ● ● ●

◇◇ ◇ ◇ ◇ ◇ ◇

◇◇ ◇

◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.9

1.0

1.1

1.2

1.3

1.4

1.5

RMSE


▲

▲ ▲ ▲ ▲▲ ▲

▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲

■

■

■

■

■■ ■

■ ■■ ■ ■ ■ ■ ■ ■ ■

●

● ●

● ●●

● ● ●●

● ● ● ● ● ● ●

◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.00

0.05

0.10

0.15

0.20

0.25

0.30share

▲▲▲▲▲ ▲ ▲ ▲ ▲ ▲ ▲

▲ ▲ ▲ ▲ ▲ ▲

■

■■■■ ■

■ ■ ■ ■ ■ ■■ ■ ■ ■ ■

●

●●

●●

●

● ● ●●

● ● ● ● ● ● ●

◇

◇◇ ◇

◇ ◇◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.05

0.10

0.15

0.20

0.25

Bshare

▲ ▲

▲ ▲▲ ▲ ▲ ▲

▲▲ ▲ ▲ ▲

▲ ▲ ▲ ▲

■

■■

■■

■■

■

■ ■

■ ■ ■■ ■

■ ■

●

●

●

●●

● ●● ● ● ● ●

●● ● ● ●

◇ ◇◇

◇◇ ◇

◇

◇◇

◇ ◇ ◇ ◇ ◇◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.10

0.15

0.20

0.25

HBshare

▲

▲ ▲ ▲ ▲▲ ▲

▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲

■

■

■

■

■■ ■

■ ■■ ■ ■ ■ ■ ■ ■ ■

●

● ●

● ●●

● ● ●●

● ● ● ● ● ● ●

◇

◇

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.00

0.05

0.10

0.15

0.20

0.25

0.30share


Method IndividualBayesian

Summary

• MixedIntegerProgrammingforACBCA– n-variatefunctionto2-variatefunction+MIP– AdvancedMIPformulation+solver– Easytoaccesswith!

• Alsoforotherestimatorvariance/linearmodels• Significantlyfasterreductionofestimatorvariance• Future:– JuliaPackage–MIPflexibility➔ManagerialObjective





Affiliations

Links


using julia and jumpfor personalized product ...web.mit.edu/.../ellipsoidconjoint_uai17.pdf ·...

Documents