generalized additive models · what is smoothing? how do gams work? fitting gams using dsm. what is...
TRANSCRIPT
![Page 1: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/1.jpg)
GeneralizedAdditiveModels
![Page 2: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/2.jpg)
OverviewWhatisaGAM?
Whatissmoothing?
HowdoGAMswork?
FittingGAMsusingdsm
![Page 3: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/3.jpg)
WhatisaGAM?
![Page 4: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/4.jpg)
"gam"1. Collectivenounusedtorefertoagroupofwhales,orrarelyalsoof
porpoises;apod.
2. (byextension)Asocialgatheringofwhalers(whalingships).
(viaNatalieKelly,AAD.SeeninMobyDick.)
![Page 5: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/5.jpg)
GeneralizedAdditiveModelsGeneralized:manyresponsedistributions
Additive:termsaddtogether
Models:well,it'samodel…
![Page 6: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/6.jpg)
Whatdoesamodellooklike?Count distributedaccordingtosomecountdistribution
Modelassumofterms
nj
![Page 7: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/7.jpg)
Mathematically...Takingthepreviousexample…
where , countdistribution
= exp [ + s( ) + s( )] +nj Aj p̂j β0 yj Depthj ϵj
∼ N(0, )ϵj σ2 ∼nj
areaofsegment-offsetprobabilityofdetectioninsegmentlinkfunctionmodelterms
![Page 8: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/8.jpg)
Response
where ,
= exp[ + s( ) + s( )] +nj Aj p̂j β0 yj Depthj ϵj
∼ N(0, )ϵj σ2 ∼ countdistributionnj
![Page 9: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/9.jpg)
CountdistributionsResponseisacount(notnotalwaysinteger)
Often,it'smostlyzero(that'scomplicated)
Wantresponsedistributionthatdealswiththat
Flexiblemean-variancerelationship
![Page 10: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/10.jpg)
Tweediedistribution
Commondistributionsaresub-cases:
Poisson
Gamma
Normal
Weareinterestedin
(here )
Var (count) = ϕE(count)
q = 1 ⇒q = 2 ⇒q = 3 ⇒
1 < q < 2q = 1.2, 1.3,…,1.9
![Page 11: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/11.jpg)
Negativebinomialdistribution
Estimate
Isquadraticrelationshipa“strong”assumption?
SimilartoPoisson:
Var (count) =E(count) + κE(count)2
κ
Var (count) = E(count)
![Page 12: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/12.jpg)
Smoothterms
where , countdistribution
= exp[ + s( ) + s( )] +nj Aj p̂j β0 yj Depthj ϵj
∼ N(0, )ϵj σ2 ∼nj
![Page 13: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/13.jpg)
Okay,butwhataboutthese"s"things?Think =smooth
Wanttomodelthecovariatesflexibly
Covariatesandresponsenotnecessarilylinearlyrelated!
Wantsomewiggles
s
![Page 14: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/14.jpg)
Whatissmoothing?
![Page 15: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/15.jpg)
Straightlinesvs.interpolationWantalinethatis“close”toallthedata
Don'twantinterpolation–weknowthereis“error”
Balancebetweeninterpolationand“fit”
![Page 16: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/16.jpg)
SplinesFunctionsmadeofother,simplerfunctions
Basisfunctions ,estimate
Makesthemathsmucheasier
bk
βk
s(x) = (x)∑Kk=1 βkbk
![Page 17: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/17.jpg)
MeasuringwigglynessVisually:
Lotsofwiggles==NOTSMOOTH
Straightline==VERYSMOOTH
Howdowedothismathematically?
Derivatives!
(Calculuswasausefulclassafterall)
![Page 18: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/18.jpg)
Wigglynessbyderivatives
![Page 19: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/19.jpg)
MakingwigglynessmatterIntegrationofderivative(squared)giveswigglyness
Fitneedstobepenalised
Penaltymatrixgivesthewigglyness
Estimatethe termsbutpenaliseobjective
“closenesstodata”+penalty
βk
![Page 20: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/20.jpg)
PenaltymatrixForeach calculatethepenalty
Penaltyisafunctionof
calculatedonce
smoothingparameter( )dictatesinfluence
bk
β
λ SββT
S
λ
![Page 21: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/21.jpg)
Smoothingparameter
![Page 22: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/22.jpg)
Howwigglyarethings?Wecansetbasiscomplexityor“size”( )
Maximumwigglyness
Smoothshaveeffectivedegreesoffreedom(EDF)
EDF<
Set “largeenough”
k
k
k
![Page 23: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/23.jpg)
WhyGAMsarecool...Fancysmooths(cyclic,boundaries,…)
Fancyresponses(expfamilyandbeyond!)
Randomeffects(byequivalence)
Markovrandomfields
Correlationstructures
SeeWood(2006/2017)forahandyintro
![Page 24: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/24.jpg)
Okay,thatwasalotoftheory...
![Page 25: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/25.jpg)
Exampledata
![Page 26: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/26.jpg)
Exampledata
![Page 27: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/27.jpg)
Exampledata
![Page 28: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/28.jpg)
SpermwhalesofftheUSeastcoastHangoutnearcanyons,eatsquid
Surveysin2004,USeastcoast
Combinationofdatafrom2NOAAcruises
ThankstoDebiPalka(NOAANEFSC),LanceGarrison(NOAASEFSC)fordata.JasonRoberts(DukeUniversity)fordataprep.
![Page 29: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/29.jpg)
ModelformulationPurespatial,pureenvironmental,mixed?
Mayhavesomepriorknowledge
Biology/ecology
Whataredriversofdistribution?
Inferentialaim
Abundance
Ecology
![Page 30: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/30.jpg)
FittingGAMsusingdsm
![Page 31: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/31.jpg)
TranslatingmathsintoR
where , countdistribution
insidethelink:formula=count ~ s(y)responsedistribution:family=nb()orfamily=tw()detectability:ddf.obj=df_hroffset,data:segment.data=segs, observation.data=obs
= exp[ + s( )] +nj Aj p̂j β0 yj ϵj
∼ N(0, )ϵj σ2 ∼nj
![Page 32: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/32.jpg)
YourfirstDSM
dsmisbasedonmgcvbySimonWood
library(dsm)dsm_x_tw <- dsm(count~s(x), ddf.obj=df, segment.data=segs, observation.data=obs, family=tw())
![Page 33: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/33.jpg)
Whatdidthatdo?summary(dsm_x_tw)
Family: Tweedie(p=1.326) Link function: log
Formula:count ~ s(x) + offset(off.set)
Parametric coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -19.8115 0.2277 -87.01 <2e-16 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Approximate significance of smooth terms: edf Ref.df F p-value s(x) 4.962 6.047 6.403 1.07e-06 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-sq.(adj) = 0.0283 Deviance explained = 17.7%-REML = 409.94 Scale est. = 6.0413 n = 949
![Page 34: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/34.jpg)
Plottingplot(dsm_x_tw)Dashedlinesindicate+/-2standarderrors
Rugplot
Onthelinkscale
EDFon axisy
![Page 35: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/35.jpg)
AddingatermJustuse+
dsm_xy_tw <- dsm(count ~ s(x) + s(y), ddf.obj=df, segment.data=segs, observation.data=obs, family=tw())
![Page 36: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/36.jpg)
Summarysummary(dsm_xy_tw)
Family: Tweedie(p=1.306) Link function: log
Formula:count ~ s(x) + s(y) + offset(off.set)
Parametric coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -20.0908 0.2381 -84.39 <2e-16 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Approximate significance of smooth terms: edf Ref.df F p-value s(x) 4.943 6.057 3.224 0.004239 ** s(y) 5.293 6.420 4.034 0.000322 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-sq.(adj) = 0.0678 Deviance explained = 27.3%-REML = 399.84 Scale est. = 5.3157 n = 949
![Page 37: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/37.jpg)
Plotting
scale=0:eachplotondifferentscalepages=1:plottogether
plot(dsm_xy_tw, pages=1)
![Page 38: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/38.jpg)
BivariatetermsAssumedanadditivestructure
Nointeraction
Wecanspecifys(x,y)(ands(x,y,z,...))
![Page 39: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/39.jpg)
ThinplateregressionsplinesDefaultbasis
Onebasisfunctionperdatapoint
Reduce#basisfunctions(eigendecomposition)
Fittingonreducedproblem
Multidimensional
![Page 40: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/40.jpg)
Thinplatesplines(2-D)
![Page 41: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/41.jpg)
Bivariatespatialtermdsm_xyb_tw <- dsm(count ~ s(x, y), ddf.obj=df, segment.data=segs, observation.data=obs, family=tw())
![Page 42: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/42.jpg)
Summarysummary(dsm_xyb_tw)
Family: Tweedie(p=1.29) Link function: log
Formula:count ~ s(x, y) + offset(off.set)
Parametric coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -20.2745 0.2477 -81.85 <2e-16 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Approximate significance of smooth terms: edf Ref.df F p-value s(x,y) 16.89 21.12 4.333 3.73e-10 ***---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R-sq.(adj) = 0.102 Deviance explained = 34.6%-REML = 394.86 Scale est. = 4.8248 n = 949
![Page 43: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/43.jpg)
Plotting...erm...plot(dsm_xyb_tw)
![Page 44: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/44.jpg)
Let'strysomethingdifferent
Stillonlinkscale
too.farexcludespointsfarfromdata
plot(dsm_xyb_tw, select=1, scheme=2, asp=1)
![Page 45: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/45.jpg)
Comparingbivariateandadditivemodels
![Page 46: Generalized Additive Models · What is smoothing? How do GAMs work? Fitting GAMs using dsm. What is a GAM? "gam" 1. Collective noun used to refer to a group of whales, or rarely also](https://reader034.vdocuments.site/reader034/viewer/2022052017/602feef2d329b95acb668502/html5/thumbnails/46.jpg)
Let'shaveago...