lecture 2 - diagnostics and model evaluationcr173/sta444_fa18/slides/lec02/lec02.pdf · ## 3...
TRANSCRIPT
![Page 1: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/1.jpg)
Lecture 2Diagnostics and Model Evaluation
Colin Rundel10/05/2018
1
![Page 2: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/2.jpg)
Some more linear models
![Page 3: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/3.jpg)
Linear model and data
ggplot(d, aes(x=x,y=y)) +geom_line() +geom_smooth(method=”lm”, color=”blue”, se = FALSE)
−2.5
0.0
2.5
5.0
7.5
0 25 50 75 100
x
y
2
![Page 4: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/4.jpg)
Linear model
l = lm(y ~ x, data=d)
summary(l)#### Call:## lm(formula = y ~ x, data = d)#### Residuals:## Min 1Q Median 3Q Max## -2.6041 -1.2142 -0.1973 1.1969 3.7072#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) -1.030315 0.310326 -3.32 0.00126 **## x 0.068409 0.005335 12.82 < 2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 1.54 on 98 degrees of freedom## Multiple R-squared: 0.6266, Adjusted R-squared: 0.6227## F-statistic: 164.4 on 1 and 98 DF, p-value: < 2.2e-16
3
![Page 5: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/5.jpg)
Bayesian model specification (JAGS)
model =”model{# Likelihoodfor(i in 1:length(y)){y[i] ~ dnorm(mu[i], tau)mu[i] = beta[1] + beta[2]*x[i]
}
# Prior for betafor(j in 1:2){beta[j] ~ dnorm(0,1/100)
}
# Prior for sigma / tau2tau ~ dgamma(1, 1)sigma2 = 1/tau
}”
4
![Page 6: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/6.jpg)
Bayesian model fitting (JAGS)
n_burn = 1000; n_iter = 5000
m = rjags::jags.model(textConnection(model), data=d,quiet=TRUE, n.chains = 4
)update(m, n.iter=n_burn, progress.bar=”none”)samp = rjags::coda.samples(m, variable.names=c(”beta”,”sigma2”),n.iter=n_iter, progress.bar=”none”
)
str(samp, max.level=1)## List of 4## $ : 'mcmc' num [1:5000, 1:3] -0.747 -0.677 -0.911 -0.906 -0.794 ...## ..- attr(*, ”dimnames”)=List of 2## ..- attr(*, ”mcpar”)= num [1:3] 1001 6000 1## $ : 'mcmc' num [1:5000, 1:3] -1.57 -1.55 -1.55 -1.69 -1.32 ...## ..- attr(*, ”dimnames”)=List of 2## ..- attr(*, ”mcpar”)= num [1:3] 1001 6000 1## $ : 'mcmc' num [1:5000, 1:3] -0.981 -1.087 -0.768 -0.831 -0.907 ...## ..- attr(*, ”dimnames”)=List of 2## ..- attr(*, ”mcpar”)= num [1:3] 1001 6000 1## $ : 'mcmc' num [1:5000, 1:3] -0.67 -0.854 -0.959 -1.169 -1.375 ...## ..- attr(*, ”dimnames”)=List of 2## ..- attr(*, ”mcpar”)= num [1:3] 1001 6000 1## - attr(*, ”class”)= chr ”mcmc.list” 5
![Page 7: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/7.jpg)
tidybayes
df_mcmc = tidybayes::gather_draws(samp, beta[i], sigma2) %>%mutate(parameter = paste0(.variable, ifelse(is.na(i),””,paste0(”[”,i,”]”)))) %>%group_by(parameter, .chain)
head(df_mcmc)## # A tibble: 6 x 7## # Groups: parameter, .chain [2]## .chain .iteration .draw i .variable .value parameter## <int> <int> <int> <int> <chr> <dbl> <chr>## 1 1 1 1 1 beta -0.747 beta[1]## 2 1 1 1 2 beta 0.0646 beta[2]## 3 1 2 2 1 beta -0.677 beta[1]## 4 1 2 2 2 beta 0.0581 beta[2]## 5 1 3 3 1 beta -0.911 beta[1]## 6 1 3 3 2 beta 0.0625 beta[2]
tail(df_mcmc)## # A tibble: 6 x 7## # Groups: parameter, .chain [1]## .chain .iteration .draw i .variable .value parameter## <int> <int> <int> <int> <chr> <dbl> <chr>## 1 4 4995 19995 NA sigma2 1.69 sigma2## 2 4 4996 19996 NA sigma2 2.43 sigma2## 3 4 4997 19997 NA sigma2 2.20 sigma2## 4 4 4998 19998 NA sigma2 2.65 sigma2## 5 4 4999 19999 NA sigma2 1.81 sigma2## 6 4 5000 20000 NA sigma2 2.43 sigma2 6
![Page 8: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/8.jpg)
Posterior plots
ggplot(df_mcmc,aes(fill=as.factor(.chain), group=.chain, x=.value)) +geom_density(alpha=0.5, color=NA) +facet_wrap(~parameter, scales = ”free”)
beta[1] beta[2] sigma2
−2.0 −1.5 −1.0 −0.5 0.0 0.05 0.06 0.07 0.08 0.09 2 3 4
0.00
0.25
0.50
0.75
1.00
1.25
0
20
40
60
80
0.0
0.5
1.0
.value
dens
ity
as.factor(.chain)
1
2
3
4
7
![Page 9: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/9.jpg)
Trace plots
df_mcmc %>% filter(.iteration %% 10 == 0) %>%ggplot(aes(x=.iteration, y=.value, color=as.factor(.chain))) +
geom_line(alpha=0.5) +facet_grid(parameter~.chain, scale=”free_y”) +geom_smooth(method=”loess”) + labs(color=”chain”)
1 2 3 4
beta[1]beta[2]
sigma2
0 10002000300040005000 0 10002000300040005000 0 10002000300040005000 0 10002000300040005000
−2.0
−1.5
−1.0
−0.5
0.0
0.05
0.06
0.07
0.08
1.52.02.53.03.54.0
.iteration
.val
ue
chain
1
2
3
4
8
![Page 10: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/10.jpg)
Credible Intervals
df_ci = tidybayes::mean_hdi(df_mcmc, .value, .width=c(0.8, 0.95))
df_ci## # A tibble: 24 x 8## # Groups: parameter [3]## parameter .chain .value .lower .upper .width .point .interval## <chr> <int> <dbl> <dbl> <dbl> <dbl> <chr> <chr>## 1 beta[1] 1 -1.01 -1.42 -0.635 0.8 mean hdi## 2 beta[1] 2 -1.03 -1.42 -0.627 0.8 mean hdi## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi## 5 beta[2] 1 0.0680 0.0614 0.0748 0.8 mean hdi## 6 beta[2] 2 0.0683 0.0612 0.0748 0.8 mean hdi## 7 beta[2] 3 0.0685 0.0613 0.0751 0.8 mean hdi## 8 beta[2] 4 0.0684 0.0613 0.0752 0.8 mean hdi## 9 sigma2 1 2.39 1.91 2.78 0.8 mean hdi## 10 sigma2 2 2.39 1.92 2.78 0.8 mean hdi## # ... with 14 more rows
9
![Page 11: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/11.jpg)
Aside - mean_qi vs mean_hdi
These differ in the use of the quantile interval vs. the highest-densityinterval.
dist_1 dist_2hpd
qi
0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 2.0
0
1
2
3
4
0
1
2
3
4
x
dens
ity
prob
0.5
0.95
10
![Page 12: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/12.jpg)
Aside - mean_qi vs mean_hdi
These differ in the use of the quantile interval vs. the highest-densityinterval.
dist_1 dist_2hpd
qi
0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 2.0
0
1
2
3
4
0
1
2
3
4
x
dens
ity
prob
0.5
0.95
10
![Page 13: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/13.jpg)
Caterpillar Plots
df_ci %>%ggplot(aes(x=.value, y=.chain, color=as.factor(.chain))) +facet_grid(parameter~.) +tidybayes::geom_pointintervalh() +ylim(0.5,4.5)
beta[1]beta[2]
sigma2
−1 0 1 2 3
1
2
3
4
1
2
3
4
1
2
3
4
.value
.cha
in
11
![Page 14: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/14.jpg)
Prediction
![Page 15: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/15.jpg)
Revised model
model_pred =”model{# Likelihoodfor(i in 1:length(y)){mu[i] = beta[1] + beta[2]*x[i]y[i] ~ dnorm(mu[i], tau)y_pred[i] ~ dnorm(mu[i], tau)
}
# Prior for betafor(j in 1:2){beta[j] ~ dnorm(0,1/100)
}
# Prior for sigma / tau2tau ~ dgamma(1, 1)sigma2 = 1/tau
}”
12
![Page 16: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/16.jpg)
Revised fitting
n_burn = 1000; n_iter = 5000
m = rjags::jags.model(textConnection(model_pred), data=d,quiet=TRUE, n.chains = 1
)
update(m, n.iter=n_burn, progress.bar=”none”)
pred = rjags::coda.samples(m, variable.names=c(”beta”,”sigma2”,”mu”,”y_pred”,”y”,”x”),n.iter=n_iter, progress.bar=”none”
)
13
![Page 17: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/17.jpg)
Predictions
df_pred = tidybayes::spread_draws(pred, y_pred[i], y[i], x[i], mu[i]) %>%mutate(resid = y - mu)
df_pred## # A tibble: 500,000 x 9## # Groups: i [100]## .chain .iteration .draw i y_pred y x mu resid## <int> <int> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 1 1 1 1 -1.67 -3.24 1 -0.825 -2.42## 2 1 2 2 1 0.340 -3.24 1 -0.984 -2.26## 3 1 3 3 1 -0.554 -3.24 1 -0.800 -2.44## 4 1 4 4 1 -3.26 -3.24 1 -0.727 -2.51## 5 1 5 5 1 0.396 -3.24 1 -0.718 -2.52## 6 1 6 6 1 0.172 -3.24 1 -0.696 -2.54## 7 1 7 7 1 -1.24 -3.24 1 -0.797 -2.44## 8 1 8 8 1 -0.251 -3.24 1 -0.736 -2.50## 9 1 9 9 1 -0.829 -3.24 1 -0.863 -2.38## 10 1 10 10 1 -0.503 -3.24 1 -0.573 -2.67## # ... with 499,990 more rows
14
![Page 18: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/18.jpg)
𝜇 vs 𝑦𝑝𝑟𝑒𝑑
i=60 i=80 i=100
i=1 i=20 i=40
−8 −4 0 4 8 12−8 −4 0 4 8 12−8 −4 0 4 8 12
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
y
dens
ity
param
mu
y_pred
15
![Page 19: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/19.jpg)
Predictions
df_pred %>% ungroup() %>% filter(.iteration %% 10 == 0) %>%ggplot(aes(x=x)) +
geom_line(aes(y=mu, group=.iteration), color=”red”, alpha=0.05) +geom_point(data=d, aes(y=y))
−2.5
0.0
2.5
5.0
7.5
0 25 50 75 100
x
mu
16
![Page 20: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/20.jpg)
Posterior distribution (𝜇)
df_pred %>% ungroup() %>%ggplot(aes(x=x)) +tidybayes::stat_lineribbon(aes(y=mu), alpha=0.5) +geom_point(data=d, aes(y=y))
−2.5
0.0
2.5
5.0
7.5
0 25 50 75 100
x
mu
0.95
0.8
0.5
17
![Page 21: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/21.jpg)
Posterior predictive distribution (𝑦𝑝𝑟𝑒𝑑)
df_pred %>% ungroup() %>%ggplot(aes(x=x)) +tidybayes::stat_lineribbon(aes(y=y_pred), alpha=0.5) +geom_point(data=d, aes(y=y))
0
5
0 25 50 75 100
x
y_pr
ed
0.95
0.8
0.5
18
![Page 22: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/22.jpg)
Residual plot
df_pred %>% ungroup() %>%ggplot(aes(x=x, y=resid)) +geom_boxplot(aes(group=x), outlier.alpha = 0.2)
−2
0
2
4
0 25 50 75 100
x
resi
d
19
![Page 23: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/23.jpg)
Model Evaluation
![Page 24: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/24.jpg)
Model assessment?
If we think back to our first regression class, one common option is 𝑅2
which gives us the variability in 𝑌 explained by our model.
Quick review:
𝑛∑𝑖=1
(𝑌𝑖 − ̄𝑌 )2
Total
=𝑛
∑𝑖=1
( ̂𝑌𝑖 − ̄𝑌 )2
Model
+𝑛
∑𝑖=1
(𝑌𝑖 − ̂𝑌𝑖)2
Error
𝑅2 = 𝑆𝑆𝑚𝑜𝑑𝑒𝑙𝑆𝑆𝑡𝑜𝑡𝑎𝑙
=∑𝑛
𝑖=1 ( ̂𝑌𝑖 − ̄𝑌 )2
∑𝑛𝑖=1 (𝑌𝑖 − ̄𝑌 )2 = Var(�̂�)
Var(𝐘) = Cor(𝐘, �̂�)2
20
![Page 25: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/25.jpg)
Model assessment?
If we think back to our first regression class, one common option is 𝑅2
which gives us the variability in 𝑌 explained by our model.
Quick review:
𝑛∑𝑖=1
(𝑌𝑖 − ̄𝑌 )2
Total
=𝑛
∑𝑖=1
( ̂𝑌𝑖 − ̄𝑌 )2
Model
+𝑛
∑𝑖=1
(𝑌𝑖 − ̂𝑌𝑖)2
Error
𝑅2 = 𝑆𝑆𝑚𝑜𝑑𝑒𝑙𝑆𝑆𝑡𝑜𝑡𝑎𝑙
=∑𝑛
𝑖=1 ( ̂𝑌𝑖 − ̄𝑌 )2
∑𝑛𝑖=1 (𝑌𝑖 − ̄𝑌 )2 = Var(�̂�)
Var(𝐘) = Cor(𝐘, �̂�)2
20
![Page 26: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/26.jpg)
Model assessment?
If we think back to our first regression class, one common option is 𝑅2
which gives us the variability in 𝑌 explained by our model.
Quick review:
𝑛∑𝑖=1
(𝑌𝑖 − ̄𝑌 )2
Total
=𝑛
∑𝑖=1
( ̂𝑌𝑖 − ̄𝑌 )2
Model
+𝑛
∑𝑖=1
(𝑌𝑖 − ̂𝑌𝑖)2
Error
𝑅2 = 𝑆𝑆𝑚𝑜𝑑𝑒𝑙𝑆𝑆𝑡𝑜𝑡𝑎𝑙
=∑𝑛
𝑖=1 ( ̂𝑌𝑖 − ̄𝑌 )2
∑𝑛𝑖=1 (𝑌𝑖 − ̄𝑌 )2 = Var(�̂�)
Var(𝐘) = Cor(𝐘, �̂�)2
20
![Page 27: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/27.jpg)
Bayesian 𝑅2
When we compute any statistic for our model we want to do so at eachiteration so that we can obtain the posterior distribution of that particularstatistic (e.g. the posterior distribution of 𝑅2 in this case).
df_R2 = df_pred %>%group_by(.iteration) %>%summarize(R2_classic = var(mu) / var(y),R2_bayes = var(mu) / (var(mu) + var(resid))
)
df_R2## # A tibble: 5,000 x 3## .iteration R2_classic R2_bayes## <int> <dbl> <dbl>## 1 1 0.622 0.625## 2 2 0.569 0.603## 3 3 0.587 0.611## 4 4 0.584 0.609## 5 5 0.560 0.599## 6 6 0.520 0.579## 7 7 0.504 0.570## 8 8 0.610 0.620## 9 9 0.592 0.613## 10 10 0.575 0.605## # ... with 4,990 more rows
21
![Page 28: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/28.jpg)
Uh oh …
summary(df_R2$R2_classic)## Min. 1st Qu. Median Mean 3rd Qu. Max.## 0.3059 0.5614 0.6271 0.6288 0.6930 1.0211
summary(df_R2$R2_bayes)## Min. 1st Qu. Median Mean 3rd Qu. Max.## 0.4155 0.5994 0.6268 0.6212 0.6488 0.7079
22
![Page 29: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/29.jpg)
Visually
df_R2 %>% tidyr::gather(method, R2, -.iteration) %>%ggplot(aes(x=R2, fill=method)) +
geom_density(alpha=0.5) +geom_vline(xintercept=modelr::rsquare(l, d), size=1)
0
3
6
9
12
0.4 0.6 0.8 1.0
R2
dens
ity
method
R2_bayes
R2_classic
23
![Page 30: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/30.jpg)
What if we collapsed first?
df_pred %>%group_by(i) %>%summarize(mu = mean(mu), y=mean(y), resid=mean(resid)) %>%summarize(R2_classic = var(mu) / var(y),R2_bayes = var(mu) / (var(mu) + var(resid))
)## # A tibble: 1 x 2## R2_classic R2_bayes## <dbl> <dbl>## 1 0.625 0.626
modelr::rsquare(l, data=d)## [1] 0.6265565
24
![Page 31: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/31.jpg)
Some problems with 𝑅2
Some new issues,
• 𝑅2 doesn’t really make sense in the Bayesian context
• multiple possible definitions with different properties• fundamental equality doesn’t hold anymore
Some old issues,
• 𝑅2 always increases (or stays the same) when a predictor is added
• 𝑅2 is highly susceptible to over fitting
• 𝑅2 is sensitive to outliers
• 𝑅2 depends heavily on current values of 𝑌• 𝑅2 can differ drastically for two equivalent models (i.e. nearlyidentical inferences about key parameters)
25
![Page 32: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/32.jpg)
Some problems with 𝑅2
Some new issues,
• 𝑅2 doesn’t really make sense in the Bayesian context
• multiple possible definitions with different properties• fundamental equality doesn’t hold anymore
Some old issues,
• 𝑅2 always increases (or stays the same) when a predictor is added
• 𝑅2 is highly susceptible to over fitting
• 𝑅2 is sensitive to outliers
• 𝑅2 depends heavily on current values of 𝑌• 𝑅2 can differ drastically for two equivalent models (i.e. nearlyidentical inferences about key parameters)
25
![Page 33: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/33.jpg)
Some Other Metrics
![Page 34: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/34.jpg)
Root Mean Square Error
The traditional definition of rmse is as follows
RMSE = √ 1𝑛
𝑛∑𝑖=1
(𝑌𝑖 − ̂𝑌𝑖)2
In the bayesian context, we have posterior samples from each parameter /prediction of interest so we can express this as
1𝑚
𝑚∑𝑠=1
√ 1𝑛
𝑛∑𝑖=1
(𝑌𝑖 − ̂𝑌 𝑠𝑖 )2
where 𝑚 is the number of iterations and ̂𝑌 𝑠𝑖 is the prediction for 𝑌𝑖 at
iteration 𝑠.
26
![Page 35: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/35.jpg)
Root Mean Square Error
The traditional definition of rmse is as follows
RMSE = √ 1𝑛
𝑛∑𝑖=1
(𝑌𝑖 − ̂𝑌𝑖)2
In the bayesian context, we have posterior samples from each parameter /prediction of interest so we can express this as
1𝑚
𝑚∑𝑠=1
√ 1𝑛
𝑛∑𝑖=1
(𝑌𝑖 − ̂𝑌 𝑠𝑖 )2
where 𝑚 is the number of iterations and ̂𝑌 𝑠𝑖 is the prediction for 𝑌𝑖 at
iteration 𝑠.
26
![Page 36: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/36.jpg)
Continuous Rank Probability Score
Another approach is the continuous rank probability score which comesfrom the probabilistic forecasting literature, it compares the full posteriorpredictive distribution to the observation / truth.
CRPS = ∫∞
−∞(𝐹 ̂𝑌 (𝑧) − 1𝑧≥𝑌 )2 𝑑𝑧
where 𝐹 ̂𝑌 is thes CDF of ̂𝑌 (the posterior predictive distribution for 𝑌 ) and1𝑧≥𝑌 is the indicator function which equals 1 when 𝑧 ≥ 𝑌 , thetrue/observed value of 𝑌 .
Since this calculates a score for a single probabilistic prediction we cannaturally extend it to multiple predictions by calculating an average CRPS
1𝑛
𝑛∑𝑖=1
∫∞
−∞(𝐹 ̂𝑌𝑖
(𝑧) − 1𝑧≥𝑌𝑖)2 𝑑𝑧
27
![Page 37: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/37.jpg)
Continuous Rank Probability Score
Another approach is the continuous rank probability score which comesfrom the probabilistic forecasting literature, it compares the full posteriorpredictive distribution to the observation / truth.
CRPS = ∫∞
−∞(𝐹 ̂𝑌 (𝑧) − 1𝑧≥𝑌 )2 𝑑𝑧
where 𝐹 ̂𝑌 is thes CDF of ̂𝑌 (the posterior predictive distribution for 𝑌 ) and1𝑧≥𝑌 is the indicator function which equals 1 when 𝑧 ≥ 𝑌 , thetrue/observed value of 𝑌 .
Since this calculates a score for a single probabilistic prediction we cannaturally extend it to multiple predictions by calculating an average CRPS
1𝑛
𝑛∑𝑖=1
∫∞
−∞(𝐹 ̂𝑌𝑖
(𝑧) − 1𝑧≥𝑌𝑖)2 𝑑𝑧
27
![Page 38: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/38.jpg)
CDF vs Indicator
0.00
0.25
0.50
0.75
1.00
−10 −5 0 5 10
value
y
28
![Page 39: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/39.jpg)
Empirical CDF vs Indicator
0.00
0.25
0.50
0.75
1.00
−5.0 −2.5 0.0 2.5 5.0
value
y
29
![Page 40: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/40.jpg)
Accuracy vs. Precision
dist1 (rmse = 1.999) dist2 (rmse = 2.821) dist3 (rmse = 1.003) dist4 (rmse = 2.224)
−5 0 5 −5 0 5 −5 0 5 −5 0 50.0
0.1
0.2
0.3
0.4
value
dens
ity
dist
dist1
dist2
dist3
dist4
dist1 (crps = 0.461) dist2 (crps = 1.192) dist3 (crps = 0.234) dist4 (crps = 1.449)
−10 −5 0 5 10 −10 −5 0 5 10 −10 −5 0 5 10 −10 −5 0 5 100.00
0.25
0.50
0.75
1.00
value
y
dist
dist1
dist2
dist3
dist4
30
![Page 41: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/41.jpg)
Empirical Coverage
One final method, which assesses model calibration is to examine how wellcredible intervals, derived from the posterior predictive distributions of the𝑌 s, capture the true/observed values.
5
10
15
20
−4 −2 0 2
x
pred
ictio
n
0.9
0.5
31
![Page 42: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/42.jpg)
Back to our example
![Page 43: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/43.jpg)
RMSE
rmse = df_pred %>%group_by(.iteration) %>%summarize(rmse = sqrt( sum( (y - y_pred)^2 ) / n())) %>%pull(rmse)
length(rmse)## [1] 5000
head(rmse)## [1] 1.973967 1.976420 2.064705 2.181217 2.208194 2.006912
mean(rmse)## [1] 2.177006
modelr::rmse(l, data = d)## [1] 1.524528
32
![Page 44: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/44.jpg)
RMSE (𝜇)
rmse = df_pred %>%group_by(.iteration) %>%summarize(rmse = sqrt( sum( (y - mu)^2 ) / n())) %>%pull(rmse)
length(rmse)## [1] 5000
head(rmse)## [1] 1.529720 1.537993 1.526788 1.530596 1.529479 1.534994
mean(rmse)## [1] 1.540472
modelr::rmse(l, data = d)## [1] 1.524528
33
![Page 45: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/45.jpg)
CRPS
crps = df_pred %>%group_by(i) %>%summarise(crps = calc_crps(y_pred, y)) %>%pull(crps)
length(crps)## [1] 100
head(crps)## [1] 1.4972794 0.6559097 1.0840541 1.5436106 0.7118661 0.3648843
mean(crps)## [1] 0.8794432
34
![Page 46: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/46.jpg)
Empirical Coverage
df_cover = df_pred %>%group_by(x,y) %>%tidybayes::mean_hdi(y_pred, .prob = c(0.5,0.9,0.95))
df_cover %>%mutate(contains = y >= .lower & y <= .upper) %>%group_by(prob=.width) %>%summarize(emp_cov = sum(contains)/n())
## # A tibble: 3 x 2## prob emp_cov## <dbl> <dbl>## 1 0.5 0.42## 2 0.9 0.94## 3 0.95 0.97
35
![Page 47: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/47.jpg)
Posterior predictive distribution (𝑦𝑝𝑟𝑒𝑑)
df_pred %>% ungroup() %>%ggplot(aes(x=x)) +tidybayes::stat_lineribbon(aes(y=y_pred), alpha=0.5) +geom_point(data=d, aes(y=y))
0
5
0 25 50 75 100
x
y_pr
ed
0.95
0.8
0.5
36
![Page 48: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/48.jpg)
Compared to what?
![Page 49: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/49.jpg)
Polynomial fit
ggplot(d, aes(x=x,y=y)) +geom_line() +geom_smooth(method='lm', color=”blue”, se = FALSE,formula = y~poly(x,5,simple=TRUE)
)
−2
0
2
4
6
0 25 50 75 100
x
y
37
![Page 50: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/50.jpg)
Model
l_p = lm(y~poly(x,5,simple=TRUE), data=d)summary(l_p)#### Call:## lm(formula = y ~ poly(x, 5, simple = TRUE), data = d)#### Residuals:## Min 1Q Median 3Q Max## -2.63687 -0.84691 -0.06592 0.76911 2.89913#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 2.4244 0.1325 18.294 < 2e-16 ***## poly(x, 5, simple = TRUE)1 19.7471 1.3252 14.901 < 2e-16 ***## poly(x, 5, simple = TRUE)2 0.6263 1.3252 0.473 0.6376## poly(x, 5, simple = TRUE)3 7.4914 1.3252 5.653 1.68e-07 ***## poly(x, 5, simple = TRUE)4 -0.0290 1.3252 -0.022 0.9826## poly(x, 5, simple = TRUE)5 -3.2899 1.3252 -2.483 0.0148 *## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 1.325 on 94 degrees of freedom## Multiple R-squared: 0.7348, Adjusted R-squared: 0.7206## F-statistic: 52.08 on 5 and 94 DF, p-value: < 2.2e-16
38
![Page 51: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/51.jpg)
JAGS Model
poly_model =”model{# Likelihoodfor(i in 1:length(y)){y[i] ~ dnorm(mu[i], tau)y_pred[i] ~ dnorm(mu[i], tau)mu[i] = beta[1] + beta[2]*x[i] + beta[3]*x[i]^2 +
beta[4]*x[i]^3 + beta[5]*x[i]^4 + beta[6]*x[i]^5}
# Prior for betafor(j in 1:6){beta[j] ~ dnorm(0,1/1000)
}
# Prior for sigma / tau2tau ~ dgamma(1, 1)sigma2 = 1/tau
}”
39
![Page 52: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/52.jpg)
Posterior Predictive Distribution
df_poly %>% ungroup() %>%ggplot(aes(x=x)) +tidybayes::stat_lineribbon(aes(y=y_pred), alpha=0.5) +geom_point(data=d, aes(y=y))
−5
0
5
10
0 25 50 75 100
x
y_pr
ed
0.95
0.8
0.5
40
![Page 53: Lecture 2 - Diagnostics and Model Evaluationcr173/Sta444_Fa18/slides/Lec02/Lec02.pdf · ## 3 beta[1] 3 -1.03 -1.44 -0.622 0.8 mean hdi ## 4 beta[1] 4 -1.03 -1.42 -0.606 0.8 mean hdi](https://reader035.vdocuments.site/reader035/viewer/2022070212/6103af96840c8b483b760840/html5/thumbnails/53.jpg)
Comparing Results
## Joining, by = c(”parameter”, ”model”)## Joining, by = c(”parameter”, ”model”)
metric parameter y~x y~poly(x,5)
rmse mu 1.540 1.416rmse y_pred 2.177 2.003crps mu NA NAcrps y_pred 0.879 0.798emp_cov (90%) mu NA NAemp_cov (90%) y_pred 0.940 0.930
41