workshop report for 'a training workshop …data.naturalcapitalproject.org/natcap/es281_what is...
TRANSCRIPT
WORKSHOP REPORT FOR 'A TRAINING WORKSHOP ON VALUING REDUCTIONS IN
LAKE WATER NUTRIENT POLLUTION USING THE WATER QUALITY BENEFITS
SPREADSHEET'
Daniel J. Phaneuf
North Carolina State University
Version Date: 23 May 2011
This document summarizes a training workshop organized by researchers at North Carolina State
University, RTI International, Duke University, and Johns Hopkins University. The objective of the
workshop was to discuss the policy tools constructed as part of our EPA funded project Measuring
Nutrient Reduction Benefits for Policy Analysis Using Linked Non-Market Valuation and
Environmental Assessment Models. The project's overall goal was to provide an integrated
protocol for use by state water quality managers in setting numeric ambient nutrient pollution
standards for surface water. A summary of the research conducted is available in the project's
two technical documents.
The workshop was held on December 10, 2010 in Raleigh, NC. It was attended by more
than 20 people from NC and the Southeast, as well as EPA employees. Participants heard
lectures from project researchers and completed hands-on exercises using the tools we
developed. The following documents were prepared for the workshop and are included here:
Call for participants
Workshop invitation letter
Workshop Agenda
Presentations (in order given)
Case Studies
Call for Participants:
A TRAINING WORKSHOP ON VALUING REDUCTIONS IN SURFACE WATER NUTRIENT POLLUTION
December 10, 2010, 8:30am to 4:00pm
JC Raulston Arboretum, North Carolina State University, Raleigh, NC
Practical Tools for Measuring the Monetary Benefits of Better Water Quality
Presented by the Center for Environmental and
Resource Economic Policy (CEnREP) at North
Carolina State University with funding from the US
Environmental Protection Agency
Event Organizer
Dr. Daniel Phaneuf, NC State University
Why a Training Workshop? States' freshwater nutrient pollution management
objectives often involve setting ambient numeric
criteria for parameters such as total nitrogen and
chlorophyll a. Judging the appropriateness of a
particular criterion involves measuring its
monetary costs and benefits. While costs are
often known, accessible tools for benefits
assessment have historically been unavailable to
state analysts.
This workshop will present practical tools that
have been developed for use by state water
quality managers who need to conduct benefits
analysis. It represents the culmination of a grant
from the EPA's Office of Water to researchers at
NCSU, RTI International, and Duke University
to develop a protocol for this purpose.
Why Attend? Participants will hear presentations on the tools
that our team has developed, gain hands on
experience by examining case studies drawn
from their own states, and will obtain instruction
and materials needed to use the methods for their
own assessments.
Who Should Attend? The workshop is primarily designed for state and
federal government employees who are charged
with carrying out economic analyses for ambient
nutrient pollution criteria. It may also be useful
for private sector consultants who conduct these
analyses for state governments.
Project Contributors
Prof. Daniel Phaneuf, NC State University
Prof. Roger von Haefen, NC State University
Dr. Melissa Kenney, Johns Hopkins University
Dr. Carol Mansfield, RTI International
Dr. Ken Reckhow, RTI International
Dr. George Van Houtven, RTI International
Registration Information
Registration is free for state and federal
government employees who commit to
attending before November 15, 2010
Registration is $50 for state and federal
government employees after November 15,
2010.
Registration is $75 for private sector
employees.
Travel support of up to $350 per person is
available to state employees (outside of NC) on a
first come basis.
The workshop is limited to 30 participants, and
registration is on a first-come basis.
Lodging Information Hampton Inn
111 Hampton Woods Lane
Raleigh, NC 27607
919-233-1798
A block of rooms is held for our event. The rate
is $79+tax per night. Please call the hotel
directly to make reservations - refer to the NCSU
Nutrients workshop.
Please make reservations by November 15, 2010.
Contact for information and registration: Jack Crawley [email protected]
Dan Phaneuf [email protected]
Sept 17, 2010
To Whom It May Concern:
I am writing to give you advance notice about an EPA-sponsored training workshop that you or
people in your organization may be interested in attending. Its focus is on techniques for assessing
the monetary benefits of reductions in surface water nutrient pollution, with particular emphasis on
evaluating quantitative targets for criteria pollutants. This workshop represents the culmination of a
project carried out by researchers at North Carolina State University, Duke University, and RTI
International, in which the objective was to design protocols for states in the Southeast to use for
carrying out benefit analysis of their proposed nutrient standards. The day-long workshop is planned
for Friday, December 10 in Raleigh, NC at NCSU's JC Raulston Arboretum. We have modest funds
available to partially support travel to the workshop. Interested individuals should contact me (Prof.
Dan Phaneuf, [email protected], 919-515-4672) with questions, and/or to be placed on the pre-
registration list. In what follows, I will provide additional details on the project and plans for the
workshop.
In 2007 the EPA's Office of Water issued a request for proposals that would "...improve the
application of empirical methodologies to the economic valuation of the benefits from reducing
nutrient levels in the nation's water bodies." The specific objective was to aid States in their attempts
to set and evaluate numeric nutrient criteria for their water quality standards. A team led by me and
including Drs. George Van Houtven and Carol Mansfield of RTI International, Dr. Ken Reckhow of
Duke University, Dr. Melissa Kenney of Johns Hopkins University, and Dr. Roger von Haefen of
North Carolina State University, was awarded the project. Work began during the Spring of 2008.
Our goal from the start has been to conduct scientifically rigorous research, whose outputs would be
of direct use to state-level policy makers and analysts. It is these outputs that we plan to discuss -
and make available - via this training workshop.
We plan to provide instruction on two related tools at the workshop. The first is an economic model
that allows users to make predictions about how a change in a particular water body's descriptive
quality level may provide monetary benefits to visitors to the water body. The second is a statistical
model that allows users to map changes in a numeric nutrient criteria - e.g. an improvement in
chlorophyll a or ambient phosphorous levels - to the same descriptive quality levels that enter the
economic model. By using the two models together, an analyst can assess the benefits of a proposed
numeric improvement at a specific water body. Our hope is that the workshop will provide
participants with enough knowledge to make use of these tools in their evaluations of proposed
policies. Equally important, we hope that the discussion at the workshop will help us fine tune the
outputs from our project to make them more useful to state level water quality managers.
Our plan is to conduct an informal and small workshop with the group limited to perhaps 30
participants. We hope to have a sense of the makeup of participants before designing the schedule of
activities, so as to better tailor the curriculum to the needs of attendees. Nonetheless, we expect
activities will generally consist of lectures by project team members, case studies, and targeted and
open discussion. Lunch and refreshments at breaks will be provided, along with all workshop
materials.
North Carolina State University is a land-grant university and a constituent institution of The University of North Carolina
Department of Agricultural and Resource Economics Campus Box 8109 Raleigh, North Carolina 27695-8109
919.515.3107 919.515.6268 (fax)
2
Please let me know by October 8 if you or other individuals in your organization may be interested
in attending the workshop. Once I have a sense of the interest level from water quality managers and
analysts working in state governments, I will determine the extent to which we will more broadly
advertise the workshop. At that time I will also provide additional details on logistics - i.e. travel
support, accommodations for those staying overnight, and the venue location.
Please do not hesitate to contact me if you have questions. I hope our workshop is something that
you might find useful.
Sincerely,
Daniel J Phaneuf
Professor of Agricultural and Resource Economics
(919) 515-4672
A TRAINING WORKSHOP ON USING THE WATER QUALITY BENEFITS SPREADSHEET
FOR VALUING REDUCTIONS IN SURFACE WATER NUTRIENT POLLUTION
DECEMBER 10, 2010
Time Topic Presenter
8:00am – 8:30am Coffee and continental breakfast available
8:30am − 9:00am Welcome and Workshop Introduction Dan Phaneuf
NC State University
9:00am − 9:30am
Group Introductions
Each person introduces himself/herself and
discusses role in evaluating nutrient criteria
9:30am − 10am
Conceptual Basis and Protocol Overview
Dan Phaneuf
NC State University
10am − 10:15am
Break, Coffee available
10:15am − 11:45am Model Details Part I
Expert elicitation protocol Melissa Kenney
Johns Hopkins University
Water quality modeling Dan Phaneuf
NC State University
Economic modeling Roger von Haefen
NC State University
11:45am − 12:45pm Lunch, Box lunches provided
12:45pm − 1:30pm Model Details Part II
Integrated policy modeling Dan Phaneuf
NC State University
1:30pm − 2:00pm Presentation of Spreadsheet Tools Dan Phaneuf
NC State University
2:00pm – 2:15pm Break, Coffee and snacks available
2:15pm − 3:00pm Case Studies Work in smaller groups on prepared case
studies
3:00pm − 3:30pm Discussion on Case Studies/Protocol George Van Houtven
RTI International
3:30pm− 4:00pm The Bigger Picture Ken Reckhow
RTI International
4:00pm Closing Remarks Dan Phaneuf
NC State University
1
A T
RA
ININ
G W
OR
KS
HO
P O
N V
AL
UIN
G R
ED
UC
TIO
NS
IN
LA
KE
WA
TE
R
NU
TR
IEN
T P
OL
LU
TIO
N U
SIN
G T
HE
WA
TE
R Q
UA
LIT
Y B
EN
EF
ITS
SP
RE
AD
SH
EE
T
D
ecem
ber
10, 2010
Ho
sted
by:
Nort
h C
aro
lina
Sta
te U
niv
ersi
ty
Cen
ter
for
En
vir
onm
enta
l an
d R
esourc
e E
cono
mic
s P
oli
cy (
CE
nR
EP
)
RT
I In
tern
atio
nal
Fund
ed b
y U
S E
PA
Off
ice
of
Wat
er
2
A R
EQ
UE
ST
FO
R P
RO
PO
SA
LS
SP
RIN
G 2
007
".
..co
nd
uct
pro
ject
s th
at w
ill
impro
ve
the
appli
cati
on
of
emp
iric
al m
eth
od
olo
gie
s
to t
he
eco
no
mic
val
uat
ion o
f th
e ben
efit
s fr
om
red
uci
ng
nu
trie
nt
lev
els
in t
he
nat
ion
's w
ater
way
s...
"
"...
aid S
tate
s in
th
eir
atte
mpts
to e
stim
ate
mon
etar
y b
enef
its
asso
ciat
ed w
ith
nu
trie
nt
red
uct
ion
s as
they
str
ive
to a
dopt
num
eric
nu
trie
nt
crit
eria
in
to t
hei
r
Sta
te w
ater
qual
ity
sta
nd
ards.
"
"...
asse
ssm
ent
of
the
ben
efit
s as
soci
ated
wit
h n
um
eric
nu
trie
nt
stan
dar
ds
...
wou
ld d
ocu
men
t th
e d
irec
t li
nkag
e b
etw
een e
xce
ss (
nutr
ien
ts)
in t
he
wat
er a
nd
a
loss
of
eco
log
ical
ser
vic
es t
o s
oci
ety,
and p
rov
ide
a m
on
etar
y e
stim
ate
of
the
ben
efit
s fr
om
res
tori
ng
thes
e se
rvic
es."
3
PR
OJE
CT
TE
AM
PI
D
r. D
anie
l J.
Phan
euf,
Nort
h C
aro
lina
Sta
te U
niv
ersi
ty
co-P
I D
r. R
og
er v
on H
aefe
n,
Nort
h C
aro
lina
Sta
te U
niv
ersi
ty
co-P
I D
r. C
aro
l M
ansf
ield
, R
TI
Inte
rnat
ional
co-P
I D
r. G
eorg
e V
an H
outv
en,
RT
I In
tern
atio
nal
co-P
I D
r. K
en R
eck
how
, R
TI
Inte
rnat
ion
al
co-P
I D
r. M
elis
sa K
enn
ey, Jo
hns
Hopkin
s U
niv
ersi
ty
EP
A P
rog
ram
Off
icer
s M
r. T
odd D
ole
y a
nd D
r. J
uli
e H
ewit
t
Work
beg
an o
n t
he
pro
ject
Spri
ng
20
08
4
PR
OJE
CT
OB
JEC
TIV
ES
Des
ign a
nd
dep
loy
a r
igoro
us,
acc
essi
ble
, an
d t
ransf
erab
le p
roto
col
to a
llo
w:
Nee
d:
A
met
ho
d f
or
map
pin
g n
utr
ient
par
amet
er v
alues
to "
thin
gs
peo
ple
car
e
abo
ut"
A
met
ho
d f
or
map
pin
g "
thin
gs
peo
ple
car
e ab
out"
to e
con
om
ic v
alu
es.
Am
bie
nt
wa
ter
qu
ali
ty a
s g
iven
by
a
nu
mer
ic i
nd
ica
tor
(e.g
. 4
0
g/m
l
chlo
rop
hyl
l a
)
Do
lla
r-d
enom
ina
ted
aggre
ga
te b
enef
its
of
wa
ter
qu
ali
ty
impro
vem
ent
rela
tive
to a
sta
tus
qu
o.
5
PR
OJE
CT
OU
TP
UT
S
1)
A w
ater
qu
alit
y p
roduct
ion f
un
ctio
n
T
akes
am
bie
nt
mea
sure
s of
a la
ke'
s n
itro
gen
, phosp
horo
us,
ch
loro
ph
yll
a,
etc,
an
d m
aps
them
to d
escr
ipti
ve
lake
qu
alit
y c
ateg
ori
es
D
escr
ibed
in
ou
r pro
ject
's 2
009 t
echnic
al d
ocu
men
t
2)
An
eco
no
mic
mo
del
U
ses
a su
rvey
of
south
east
ern h
ou
sehold
s u
sed t
o c
alib
rate
mo
del
s m
app
ing
a la
ke'
s d
escr
ipti
ve
qual
ity l
evel
to $
-den
om
inat
ed b
enef
its
D
escr
ibed
in
ou
r pro
ject
's 2
010
tec
hnic
al d
ocu
men
t
3)
A p
roto
col
for
inte
gra
ting t
hes
e tw
o m
od
els
S
imp
le v
ersi
on
co
ded
in
to a
spre
adsh
eet
tool
allo
win
g a
nal
yst
s to
use
th
ese
met
ho
ds
for
ben
efit
s as
sess
men
t.
D
escr
ibed
in
use
r's
man
ual
and a
non
-tec
hnic
al o
ver
vie
w d
ocu
men
t.
6
WO
RK
SH
OP
OB
JEC
TIV
ES
Pre
sent
idea
s d
evel
op
ed o
ver
the
cours
e of
our
pro
ject
:
G
ive
som
e b
asic
un
der
stan
din
g o
f under
lyin
g m
od
els
(av
oid
'bla
ck b
ox
'
feel
).
D
emo
nst
rate
the
spre
adsh
eet
too
l
S
oli
cit
yo
ur
reac
tion
s to
the
pro
toco
l
G
ener
ate
dis
cuss
ion o
n t
he
met
hod a
nd p
ract
ice
of
ben
efit
s as
sess
men
t fo
r
stat
e w
ater
qu
alit
y s
tand
ards.
En
cou
rag
e an i
nfo
rmal
and p
art
icip
ato
ry a
tmo
sph
ere
7
INT
RO
DU
CT
ION
S
Ple
ase
tak
e 1
to
2 m
inu
tes
to:
1)
Intr
od
uce
you
rsel
f −
nam
e an
d o
rgan
izat
ion
2)
Des
crib
e g
ener
ally
the
work
you d
o o
n w
ater
qual
ity s
tan
dar
ds
eval
uat
ion
3)
Men
tio
n c
urr
ent
or
fort
hco
min
g e
val
uat
ion t
asks
1
CO
NC
EP
TU
AL
BA
SIS
AN
D P
RO
TO
CO
L O
VE
RV
IEW
Dan
Phan
euf
NC
SU
2
EC
ON
OM
IC C
ON
CE
PT
OF
VA
LU
E
Wh
at t
yp
es o
f ec
olo
gic
al s
ervic
es d
oes
cle
an w
ater
pro
vid
e fo
r in
div
idu
als
and
soci
etie
s?
D
rink
ing
wat
er q
ual
ity
L
and
scap
e am
enit
ies/
aest
het
ics
W
ild
life
hab
itat
R
ecre
atio
n o
pp
ort
un
itie
s
No
te t
ha
t m
an
y o
f th
ese
are
non
-mark
et −
not
boug
ht
an
d s
old
dir
ectl
y
How
do w
e co
nce
ptu
aliz
e th
e b
enef
its
of
impro
vem
ents
in
wat
er q
ual
ity
fo
r a
per
son?
C
han
ge
in a
per
son's
wel
l-bei
ng a
s s/
he
def
ines
it!
D
epen
ds
on
tas
tes,
pre
fere
nce
s, i
ndiv
idu
al c
har
acte
rist
ics
Ho
w c
an w
e m
easu
re s
om
ethin
g l
ike
this
?
3
WIL
LIN
GN
ES
S T
O P
AY
If I
am
wil
lin
g t
o p
ay X
$'s
to h
ave
som
e outc
om
e →
my
wel
l-b
eing
mu
st b
e
hig
her
wit
h t
he
ou
tco
me
and
wit
hout
the
mon
ey!
Exa
mp
le:
I w
ou
ld h
appil
y part
wit
h $
8 t
o h
ave
a 6
-pack
of
hig
h q
ua
lity
bee
r
Max
imu
m w
illi
ng
nes
s to
pay
(W
TP
) −
am
ou
nt
that
lea
ves
me
ind
iffe
ren
t
bet
wee
n (
a) h
avin
g t
he
outc
om
e an
d p
ayin
g o
r (b
) kee
pin
g t
he
mo
ney
I w
ou
ld p
ay
up t
o $
15 f
or
that
6-p
ack
bu
t n
ot
mo
re
Note
:
M
axim
um
WT
P i
s th
e doll
ar v
alue
I pla
ce o
n t
he
wel
l-b
eing
ch
ang
e fr
om
the
ou
tco
me
A
ver
y i
nd
ivid
ual
conce
pt
dep
endin
g o
n t
aste
s, p
refe
ren
ces,
ch
arac
teri
stic
s
My
sist
er i
s a
tee
tota
ler −
what
is h
er W
TP
fo
r th
e 6
-pa
ck?
4
ME
AS
UR
ING
WIL
LIN
GN
ES
S T
O P
AY
Fo
r m
ark
et g
oo
ds −
pri
ces
pro
vid
e a
good s
ense
of
peo
ple
's w
illi
ng
nes
s to
pay
Fo
r non
-mar
ket
go
od
s li
ke
wat
er q
ual
ity −
more
dif
ficu
lt
W
e g
ener
ally
do
n't
'see
' peo
ple
rea
chin
g f
or
thei
r w
alle
t an
d p
ayin
g f
or
wat
er q
ual
ity.
P
ub
lic
go
od
ch
arac
teri
stic
s (n
on
-riv
al, non-e
xcl
udab
le)
N
eed
to
be
clev
er a
nd l
ook f
or
clues
.
Loo
kin
g f
or
evid
ence
− h
ow
do
es a
per
son b
ehav
e in
an
act
ivit
y r
elat
ed t
o w
ater
qu
alit
y?
D
oes
s/h
e p
ay m
ore
for
a house
on a
lak
e if
the
wat
er i
s cl
ean
er?
D
riv
e fu
rth
er t
o v
isit
a l
ake
if t
he
wat
er q
ual
ity i
s b
ette
r?
C
on
trib
ute
les
s to
dri
nk
ing w
ater
tre
atm
ent
cost
s w
hen
po
llu
tio
n i
s lo
wer
?
Each
of
thes
e ca
n b
e li
nke
d t
o W
TP
5
RE
CR
EA
TIO
N B
EN
EF
ITS
Our
focu
s is
on t
he
recr
eati
on b
enef
its
of
impro
vin
g l
ake
wat
er q
ual
ity
:
O
uts
ide
evid
ence
sugges
ts t
his
is
typic
ally
a l
arg
e co
mp
on
ent
of
ben
efit
s
from
red
uci
ng
nu
trie
nt
po
lluti
on
N
on
-riv
al −
lots
of
geo
gra
ph
ical
ly d
isper
se p
eople
can
ben
efit
at
the
sam
e
tim
e
So
ciet
y W
illi
ngnes
s to
Pay
=
Indiv
idu
al
WT
Ps
Our
app
roac
h:
M
easu
re h
ow
mu
ch f
urt
her
peo
ple
wou
ld o
n a
ver
age
trav
el t
o e
xp
erie
nce
bet
ter
wat
er q
ual
ity
fo
r re
crea
tion
.
If p
erso
n w
ou
ld t
rave
l M
ext
ra m
iles
to e
xper
ience
qA r
ath
er t
ha
n q
B, th
ey a
re
wil
lin
g t
o p
ay
the
trave
l co
st o
f th
ose
M m
iles
to
ha
ve q
A>
qB!
6
ME
AS
UR
ED
WA
TE
R Q
UA
LIT
Y A
ND
EX
PE
RIE
NC
ED
WA
TE
R Q
UA
LIT
Y
Ass
essi
ng
wat
er q
ual
ity
− v
ia n
etw
ork
of
mon
itori
ng s
tati
on
s.
p
rov
ides
ob
ject
ive
mea
sure
s via
am
bie
nt
conce
ntr
atio
ns
of
nu
trie
nts
an
d
oth
er p
oll
uta
nts
.
Exp
erie
nci
ng
wat
er q
ual
ity −
bas
ed o
n s
enso
ry p
erce
pti
on
s an
d l
ay-l
angu
age
info
rmat
ion
.
s
eein
g w
ild
life
or
catc
hin
g f
ish
w
ater
cla
rity
, co
lor,
and s
mel
l
p
rese
nce
of
nu
isan
ce a
lgae
i
nfo
rmat
ion
on t
he
types
of
acti
vit
ies
app
ropri
ate
for
the
wat
er b
od
y
"Thin
gs
peo
ple
ca
re a
bout"
Peo
ple
are
wil
ling
to p
ay
for
wate
r quali
ty e
xper
ien
ce
Cri
teri
a a
re s
et b
ase
d o
n s
cien
tifi
c in
dic
ato
rs
7
EX
PE
RIE
NC
ED
WA
TE
R Q
UA
LIT
Y A
ND
RE
CR
EA
TIO
N B
EN
EF
ITS
A m
od
el o
f re
crea
tio
n s
ite
choic
e beh
avio
r:
P
eop
le c
ho
se a
lak
e re
crea
tion d
esti
nat
ion b
ased
on m
any
fac
tors
D
ista
nce
fro
m h
om
e (t
rav
el c
ost
) an
d l
ake
condit
ion
s ar
e am
on
g t
hem
T
rav
el i
s co
stly
− g
as a
nd d
epre
ciat
ion c
ost
s, v
alue
of
tim
e sp
ent
en r
ou
te
Qu
anti
fyin
g t
rav
el d
ista
nce
and w
ater
qual
ity t
radeo
ffs:
Tra
vel
Cost
Wat
er Q
ual
ity E
xper
ience
WT
P
Tw
o c
hal
len
ges
:
M
app
ing
mea
sure
d w
ater
qual
ity t
o e
xp
erie
nce
d w
ater
qu
alit
y
q
uan
tify
ing
lak
e v
isit
ors
wil
lingnes
s to
tra
vel
furt
her
fo
r a
bet
ter
qu
alit
y
exp
erie
nce
.
8
PR
OT
OC
OL
SC
HE
MA
TIC
1)
Use
an
ex
per
t el
icit
atio
n f
ram
ework
to l
ink m
easu
red a
nd
ex
per
ien
ced
wat
er
qu
alit
y.
B
ased
on
pri
or
wo
rk i
n N
C b
y c
o-P
Is M
elis
sa K
enn
ey a
nd
Ken
Rec
kh
ow
.
2)
Use
co
njo
int
mo
del
ing f
ram
ework
to l
inked
exp
erie
nce
wat
er q
ual
ity
to
lak
e
cho
ice
and
wil
lin
gn
ess
to t
ravel
.
B
ased
on
a s
urv
ey w
e co
ndu
cted
of
lake
use
rs i
n s
ou
thea
ster
n s
tate
s d
uri
ng
spri
ng
201
0.
Exp
ert
Elic
itat
ion
Me
lissa
A. K
en
ney
, Ph
.D.
Wh
y d
o w
e n
ee
d
exp
ert
elic
itat
ion
?
Tro
ph
ic S
tate
is a
Su
bje
ctiv
e In
dex
•C
ann
ot
be
dir
ectl
y m
easu
red
•In
dic
ated
by
mu
ltip
le v
aria
ble
s
•M
anif
ests
itse
lf d
iffe
ren
tly
in d
iffe
ren
t sy
stem
s
•N
eed
a m
eth
od
to
tra
nsp
aren
tly
qu
anti
fy
exp
ert
jud
gmen
t ab
ou
t tr
op
hic
sta
te
Wh
at is
exp
ert
e
licit
atio
n?
Exp
ert
Elic
itat
ion
•A
rig
oro
us,
sci
enti
fica
lly-d
efen
sib
le
app
roac
h t
o q
uan
tify
pro
bab
ilist
ic e
xper
t ju
dgm
ents
.
–In
terv
iew
–Su
rvey
/ W
ork
bo
ok
•R
ange
of
app
roac
hes
an
d b
est
pra
ctic
es f
or
dif
fere
nt
met
ho
ds
Exp
ert
Elic
itat
ion
Ref
eren
ces
Cle
men
, R. T
., a
nd
T. R
eilly
. 20
01
. Mak
ing
Har
d D
ecis
ion
s w
ith
Dec
isio
n T
oo
ls. D
uxb
ury
Th
om
son
Lea
rnin
g, P
acif
ic G
rove
, CA
.
Cle
men
, R. T
., a
nd
R. L
. Win
kler
. 19
90
. Un
anim
ity
and
Co
mp
rom
ise
amo
ng
Pro
bab
ility
Fo
reca
ster
s. M
anag
emen
t Sc
ien
ce 3
6:7
67
-77
9.
Kee
ney
, R. L
., a
nd
D. V
on
Win
terf
eld
t. 1
99
1. E
licit
ing
Pro
bab
iliti
es f
rom
Exp
erts
in C
om
ple
x Te
chn
ical
Pro
ble
ms.
Tra
nsa
ctio
ns
on
En
gin
eeri
ng
Man
agem
ent
38
:19
1-2
01
.
Ken
ney
, M.A
. 20
07
. W
hic
h n
utr
ien
t cr
iter
ia s
ho
ld s
tate
s an
d T
rib
es c
ho
ose
to
det
erm
ine
wat
erb
od
y im
pai
rmen
t?: U
sin
g sc
ien
ce a
nd
jud
gmen
ts t
o in
form
dec
isio
n m
akin
g.
Dis
sert
atio
n.
Du
ke U
niv
ersi
ty, N
ich
ola
s Sc
ho
ol o
f th
e En
viro
nm
ent
and
Ear
th S
cien
ces.
4
10
pp
.
Mey
er, M
. A.,
an
d J
. M. B
oo
ker.
20
01
. Elic
itin
g an
d A
nal
yzin
g Ex
per
t Ju
dgm
ent:
A P
ract
ical
G
uid
e. A
mer
ican
Sta
tist
ical
Ass
oci
atio
n a
nd
th
e So
ciet
y fo
r In
du
stri
al a
nd
Ap
plie
d
Mat
hem
atic
s, P
hila
del
ph
ia.
Mo
rgan
, M. G
., a
nd
M. H
enri
on
. 19
90
. U
nce
rtai
nty
: A g
uid
e to
dea
ling
wit
h u
nce
rtai
nty
in
qu
anti
tati
ve r
isk
and
po
licy
anal
ysis
. Cam
bri
dge
Un
iver
sity
Pre
ss, C
amb
rid
ge.
Rec
kho
w, K
. H.,
G. B
. Arh
on
dit
sis,
M. A
. Ken
ney
, L. H
ause
r, J
. Tri
bo
, C. W
u, K
. J. E
lco
ck, L
. J.
Stei
nb
erg,
C. A
. Sto
w, a
nd
S. J
. McB
rid
e. 2
00
5.
A p
red
icti
ve a
pp
roac
h t
o n
utr
ien
t cr
iter
ia. E
nvi
ron
men
tal S
cien
ce &
Tec
hn
olo
gy 3
9:2
91
3-2
91
9.
Ho
w d
id w
e c
on
du
ct
the
exp
ert
elic
itat
ion
?
Ou
tlin
e
•R
ecru
it E
xper
ts
•Se
lect
Dat
a
•D
evel
op
Elic
itat
ion
Su
rvey
•Ex
ecu
te
•C
alib
rati
on
•Ex
per
t C
om
bin
atio
n
•C
on
clu
sio
ns
Rec
ruit
Exp
erts
•Sc
ien
tist
s w
ith
loca
l/re
gio
nal
kn
ow
led
ge o
f tr
op
hic
sta
te f
or
the
wat
erb
od
y ty
pe
–A
cad
emic
–G
ove
rnm
en
t sc
ien
tist
s
–C
on
sult
ants
•D
iver
se r
ange
of
exp
erti
se t
o c
aptu
re r
ange
of
kno
wle
dge
•A
t le
ast
3 p
eop
le, i
dea
lly 5
peo
ple
–
Re
sear
ch in
dic
ates
5 d
iver
se e
xper
ts la
rgel
y ca
ptu
res
the
ran
ge e
xper
tise
•Le
tter
to
rec
ruit
exp
erts
in b
ind
er
Sele
ct D
ata
•Id
enti
fy w
ater
qu
alit
y va
riab
les
rela
ted
to
tro
ph
ic
stat
e
–w
e u
sed
TN
, TIN
, TP
, Ch
l a, S
ecch
i dep
th, t
urb
idit
y, a
nd
d
isso
lved
oxy
gen
•A
t le
ast
10
0 d
ata
row
s
–M
ore
ro
ws,
mo
re e
xper
t ti
me
–If
wan
t to
co
mp
are
acro
ss r
egio
ns
nee
d s
et o
f d
ata
that
is
th
e sa
me
(we
use
50
dat
a ro
ws)
•Fo
cus
on
ro
ws
that
pro
vid
e th
e m
ost
info
rmat
ion
–D
ata
is N
OT
rep
rese
nta
tive
Dev
elo
p E
licit
atio
n S
urv
ey
Exam
ple
in B
ind
er
•Le
tter
an
d C
on
sen
t Fo
rm
•D
irec
tio
ns
(no
te: c
ateg
ory
def
init
ion
s m
ay n
eed
to
be
refi
ned
fo
r lo
cal c
on
dit
ion
s)–
Imag
ine
10
0 d
iffe
ren
t la
kes
in t
he
XX
reg
ion
wit
h t
he
char
acte
rist
ics
spec
ifie
d b
y th
e gi
ven
dat
a ro
w.
Of
the
10
0 la
kes,
ho
w m
any
of
the
lake
s w
ou
ld y
ou
exp
ect
to
fal
l in
to e
ach
of
the
fo
llow
ing
eutr
op
hic
atio
n
cate
gori
es?
•M
ap o
f ar
ea, A
dd
itio
nal
Info
rmat
ion
(u
nit
s,
vari
able
sam
plin
g m
eth
od
)
•Ex
amp
le
•El
icit
atio
n (
no
te t
he
cate
gori
es a
re o
n t
he
op
po
site
pag
e to
ass
ure
co
nsi
sten
t d
efin
itio
ns)
Exec
ute
•P
re-s
urv
ey in
terv
iew
–W
hat
are
th
e m
ech
anis
ms
that
lead
to
eu
tro
ph
icat
ion
in t
his
reg
ion
?
–W
hat
oth
er v
aria
ble
s (n
on
-eu
tro
ph
icat
ion
) th
at a
ffec
t a
wat
erb
od
y’s
atta
inm
ent
of
des
ign
ated
use
?
•Su
rvey
/Wo
rkb
oo
k–
Wal
k th
rou
gh d
irec
tio
ns,
fro
nt
mat
eria
l
–G
o t
hro
ugh
an
exa
mp
le in
det
ail
–Le
t th
e ex
pe
rt g
o t
hro
ugh
an
exa
mp
le a
nd
sev
eral
dat
a ro
ws.
–A
fter
th
ey p
rovi
de
valu
es, a
sk t
hem
to
exp
lain
th
eir
reas
on
ing
for
the
valu
es.
If t
he
logi
c is
pro
ble
mat
ic,
exp
lain
dir
ecti
on
s an
d w
ork
th
rou
gh q
ues
tio
ns
agai
n.
–O
nce
th
e yo
u a
re c
om
fort
able
th
at t
he
exp
ert
u
nd
erst
and
s th
e ex
erci
se, a
llow
th
em t
o c
om
ple
te
ind
epe
nd
entl
y. A
sk t
hem
to
co
mp
lete
wit
hin
2 w
eeks
.
Cal
ibra
tio
n (
i.e.,
QA
/QC
)
•Su
mm
ariz
e th
e in
terv
iew
an
d p
rovi
de
the
rep
ort
to
th
e ex
per
t w
ith
in 1
wee
k
•En
ter
dat
a fr
om
th
e su
rvey
an
d t
hen
go
th
rou
gh
the
dat
a an
d n
ote
:–
Dat
a ro
ws
that
are
sim
ilar
wit
h d
iffe
ren
t ju
dgm
ents
–Ju
dgm
ents
th
at d
on
’t m
atch
wit
h e
xpec
tati
on
–A
co
up
le o
f ra
nd
om
ro
ws.
•Sc
hed
ule
fo
llow
-up
wit
hin
2 w
eeks
•Fo
r th
e d
ata
row
s, a
sk e
xper
t to
exp
lain
logi
c b
ehin
d t
hei
r ju
dgm
ents
•If
th
e ex
per
t w
ou
ld li
ke t
o c
han
ge t
hei
r o
rigi
nal
ju
dgm
ent,
do
so
(it
’s t
hei
r ju
dgm
ent!
)
•R
eco
rd r
easo
nin
g fo
r th
e va
lues
Exp
ert
Co
mb
inat
ion
•C
an c
on
sid
er e
xper
t ju
dgm
ents
in
dep
end
entl
y o
r as
an
agg
rega
te
•Eq
ual
wei
ghte
d a
vera
ge o
f al
l th
e ex
per
t ju
dgm
ents
.
–N
eed
th
e sa
me
dat
a ro
ws
•O
ther
co
mb
inat
ion
ap
pro
ach
es p
oss
ible
, b
ut
the
per
form
ance
is a
pp
roxi
mat
ely
the
sam
e as
ave
ragi
ng
Co
ncl
usi
on
s
•D
ocu
men
t ev
eryt
hin
g! K
eep
co
pie
s o
f th
e d
ata
row
s ch
ose
n, a
ny
mo
dif
icat
ion
to
th
e d
ata,
all
inte
rvie
ws,
up
dat
es t
o t
he
inte
rvie
w, a
ll el
icit
atio
n d
ata
(ori
gin
al,
calib
rate
d, c
om
bin
ed, e
tc.)
•Th
ink
abo
ut
wh
y yo
u w
ant
to u
se t
he
dat
a an
d m
ake
sure
th
e el
icit
atio
n p
rovi
des
yo
u
ever
yth
ing
you
nee
d
1
ES
TIM
AT
ING
A W
AT
ER
QU
AL
ITY
PR
OD
UC
TIO
N F
UN
CT
ION
Dan
Phan
euf
NC
SU
2
DE
FIN
ING
TH
E T
AS
K
Fo
r a
lak
e in
dex
ed k
:
Rk=
f(T
Nk,
TP
k, C
hla
k,..
.),
Rk ∊
(1,2
,3,4
,5)
wh
ere
Rk
is a
n o
rdin
al i
nd
ex d
escr
ibin
g t
he
qual
itat
ive/
des
crip
tiv
e le
vel
of
wat
er
qu
alit
y a
t la
ke
k.
Obje
ctiv
e:
Use
th
e ex
per
t el
icit
atio
n i
nfo
rmat
ion
to p
aram
eter
ize
a fu
nct
ion
f(∙
)
Rec
og
niz
e th
at t
he
link f
rom
(T
N,
TP
, C
hla
,...)
→ (
R)
is p
rob
abil
isti
c;
pro
du
ce p
red
icti
on
s fo
r th
e pro
bab
ilit
y t
hat
lak
e k
is i
n e
ach
of
the
fiv
e
cate
go
ries
.
Use
th
e p
rob
abil
itie
s to
pre
dic
t E
(Rk| T
Nk,
TP
k, C
hla
k)
3
SO
ME
SP
EC
IFIC
S
Mel
issa
's e
xp
ert
elic
itat
ion
dat
a is
of
the
form
:
We
wil
l in
terp
ret
this
to m
ean R
=3 f
or
this
exper
t fo
r th
ese
lev
els
of
the
par
amet
ers.
Nee
d a
mo
del
th
at p
aram
eter
izes
:
Pr(
)(
,,
,,
)ij
ijij
ijij
ijR
mg
TN
TP
Chla
Sec
chi
Tur
for
m=
1,...,5
.
Tab
le 3
: E
xam
ple
of
an E
xp
ert's
Res
po
nse
pa
ram
eter
TN
TIN
TP
Ch
laD
OS
ecch
iT
urb
idit
y
valu
e0.4
60.0
20.0
338
6.3
1.3
3.9
Ra
nk
ing
12
34
5
# L
ake
s0
10
50
40
0
4
SO
ME
MO
DE
LS
Exam
ple
est
imat
es f
rom
an
ord
ered
log
it m
od
el (
stan
dar
d e
rro
rs i
n p
aren
thes
es):
Tab
le 5
: F
ull
Sam
ple
Ord
ered
Logit
Model
Res
ult
s
VA
RIA
BL
EM
od
el 1
Mod
el 2
Mod
el 3
Tota
l N
itro
gen
0.4
36 (
0.3
01)
0.6
03 (
0.2
69)
1.3
35 (
0.2
75)
Tota
l In
org
an
ic N
itro
gen
0.8
73 (
0.4
94)
--
Tota
l P
hosp
oro
us
9.7
92 (
2.4
63)
10.4
02 (
2.6
06)
11.7
29 (
2.7
74)
Ch
loro
ph
yll
a0.0
76 (
0.0
12)
0.0
75 (
0.0
11)
0.0
72 (
0.0
1)
Dis
solv
ed O
xyg
en-0
.004 (
0.0
5)
--
Sec
chi
Dep
th-0
.73 (
0.1
39)
-0.7
05 (
0.1
37)
-
Tu
rbid
ity
0.0
17 (
0.0
09)
0.0
2 (
0.0
08)
0.0
35 (
0.0
1)
Cu
t 2
-1.1
12 (
0.7
64)
-1.0
07 (
0.5
93)
0.6
1 (
0.4
03)
Cu
t 3
0.5
35 (
0.5
46)
0.6
39 (
0.3
96)
2.1
46 (
0.3
51)
Cu
t 4
3.0
44 (
0.3
94)
3.1
52 (
0.4
04)
4.5
05 (
0.4
51)
Cu
t 5
6.2
64 (
0.5
61)
6.3
51 (
0.4
83)
7.6
95 (
0.5
78)
Log-l
ikel
ihood V
alu
e-1
,542.5
0-1
,544.3
7-1
,597.4
1
5
EX
AM
PL
E P
RE
DIC
TIO
NS
Jord
an L
ake,
No
rth
Car
oli
na:
2
2 m
on
ito
ring
sta
tion
rea
din
gs
obta
ined
M
edia
n p
aram
eter
val
ues
acr
oss
all
rea
din
gs
TN
T
P
CL
A
S
T
0.8
2 m
g/l
0
.07
5 m
g/l
3
3
g/l
0.6
0 m
1
2.0
0 N
TU
P
red
icti
on
s fo
r p
robab
ilit
ies:
Pr
1
Pr
2
Pr
3
Pr
4
Pr
5
0.0
1
0.0
4
0.3
5
0.5
5
0.0
5
E
xp
ecte
d i
nd
ex l
evel
= 3
.6
6
Rec
all
Mel
issa
's c
ateg
ori
es:
Cate
go
ryW
ate
r
cla
rity
Co
lor
Alg
ae
Nu
trie
nt
lev
els
Ox
yg
en
Od
or
Aq
uati
c l
ife
1E
xcelle
nt
None
Very
little
Very
low
Very
hig
hN
oV
ery
healthy,
abundant
2G
ood
Little
Little
Low
Hig
hL
ittle
Healthy, abundant
3F
air
Som
eM
odera
teM
odera
teM
odera
teL
ittle
Som
ew
hat
healthy,
abundant
4P
oor
Noticeable
Hig
hH
igh
Low
Noticeable
Unhealthy, sc
arc
e
5P
oor
Consi
dera
ble
Very
hig
hV
ery
hig
hL
ow
to n
oS
trong
off
ensi
ve
Unhealthy, sc
arc
e o
r
none p
rese
nt
Table
1: T
rophic
Sta
tus
Cate
gori
es
7
Lak
e N
orm
an, N
ort
h C
aro
lina:
2
4 m
on
ito
ring
sta
tion
rea
din
gs
obta
ined
M
edia
n p
aram
eter
val
ues
acr
oss
all
rea
din
gs
TN
T
P
CL
A
S
T
0.1
2 m
g/l
0
.02
mg/l
4
g/l
2.3
m
2.3
0 N
TU
P
red
icti
on
s fo
r p
robab
ilit
ies:
Pr
1
Pr
2
Pr
3
Pr
4
Pr
5
0.5
0
0.3
4
0.1
5
0.0
1
0.0
0
E
xp
ecte
d i
nd
ex l
evel
= 1
.67
8
US
ES
AN
D L
IMIT
AT
ION
S
All
ow
s us
to c
oll
apse
mult
iple
dim
ensi
on
, quan
tita
tive
asse
ssm
ents
in
to a
sin
gle
dim
ensi
on
qu
alit
ativ
e in
dex
P
rov
ides
ab
ilit
y t
o l
ink m
easu
red w
ater
qual
ity t
o e
xp
erie
nce
d w
ater
qu
alit
y
Lim
itat
ion
s an
d c
avea
tes:
E
xp
ert
elic
itat
ion
pro
toco
l w
as N
ort
h C
aroli
na
spec
ific
, an
d t
her
efore
con
dit
ion
al o
n N
C g
eo-p
hysi
cal
condit
ions
U
sin
g t
he
mo
del
far
fro
m o
rigin
al c
onte
xt
wil
l d
imin
ish
cre
dib
ilit
y o
f th
e
pre
dic
tio
ns!
Roger
H.
von H
aefe
n
NC
SU
1
G
oal:
moneti
ze t
he r
ecre
ati
onal benefi
ts
associa
ted w
ith w
ate
r quality
im
pro
vem
ents
M
eans:valu
ati
on s
urv
ey /
choic
e e
xperi
ments
2
O
nline K
now
ledge N
etw
ork
s s
urv
ey
Fie
lded A
pri
l 2
010
R
oughly
1,3
27 c
om
ple
ted s
urv
eys
Elicit
ati
on f
orm
at:
choic
e e
xperi
ments
3
Beach V
acati
on #
1Beach V
acati
on #
2
7
day
s
B
each
ho
use
U
nd
evel
op
ed b
each
N
C O
ute
r B
anks
Fu
ll co
st: $
3,5
00
5
day
s
Res
ort
D
evel
op
ed b
each
Fl
ori
da
Gu
lf C
oas
t
Full
cost
: $4
,50
0
4
Beach V
acati
on #
1Beach V
acati
on #
2
7
day
s
B
each
ho
use
U
nd
evel
op
ed b
each
N
C O
ute
r B
anks
Fu
ll co
st: $
3,5
00
5
day
s
Res
ort
D
evel
op
ed b
each
Fl
ori
da
Gu
lf C
oas
t
Full
cost
: $4
,50
0
5
Whic
h b
each v
acati
on w
ould
you c
hoose?
Beach V
acati
on #
1Beach V
acati
on #
2
7
day
s
B
each
ho
use
U
nd
evel
op
ed b
each
N
C O
ute
r B
anks
Fu
ll co
st: $
3,5
00
5
day
s
Res
ort
D
evel
op
ed b
each
Fl
ori
da
Gu
lf C
oas
t
Full
cost
: $4
,50
0
6
If t
hese b
each v
acati
ons w
ere
your
only
opti
ons,
would
you v
isit
one o
f th
em
or
not
take a
beach v
acati
on?
Based o
n s
tate
d c
hoic
es,
we c
an infe
r:
◦T
he m
arg
inal valu
e o
f an a
ddit
ional vacati
on d
ay
◦T
he v
alu
e o
f sta
yin
g a
t a b
each h
ouse v
ers
us a
re
sort
◦T
he v
alu
e o
f vis
itin
g a
n u
ndevelo
ped
vers
us
develo
ped b
each
◦T
he v
alu
e o
f vis
itin
g t
he O
ute
r Banks v
ers
us t
he
Gulf
Coast
7
W
hat
lake w
ate
r quality
att
ribute
s a
ffect
recre
ato
rbehavio
r?
In
oth
er
word
s,
how
do w
e t
ransla
te t
he
outp
ut
from
our
wate
r quality
model in
to
att
ribute
s r
ecre
ato
rsunders
tand a
nd v
alu
e?
8
T
rophic
sta
tus c
ate
gori
es
Cate
gory
Wate
r cla
rity
Colo
rA
lgae
Nutr
ient
levels
Oxygen
Od
or
Aq
uati
clife
1Excellent
None
Very
lit
tle
Very
low
Very
hig
hN
oV
ery
healt
hy
2G
ood
Lit
tle
Lit
tle
Low
Hig
hLit
tle
Healt
hy
3Fair
Som
eM
odera
teM
odera
teM
odera
teLit
tle
Som
ew
hat
healt
hy
4Poor
Noti
ce-
able
Hig
hH
igh
Low
Noti
ce-
able
Unhealt
hy
5Poor
Consid
er-
able
Very
hig
hV
ery
hig
hLow
to n
oStr
ong
off
ensiv
eU
nhealt
hy
9
T
rophic
sta
tus c
ate
gori
es
Cate
gory
Wate
r cla
rity
Colo
rA
lgae
Nutr
ient
levels
Oxygen
Od
or
Aq
uati
clife
1Excellent
None
Very
lit
tle
Very
low
Very
hig
hN
oV
ery
healt
hy
2G
ood
Lit
tle
Lit
tle
Low
Hig
hLit
tle
Healt
hy
3Fair
Som
eM
odera
teM
odera
teM
odera
teLit
tle
Som
ew
hat
healt
hy
4Poor
Noti
ceable
Hig
hH
igh
Low
Noti
ceable
Unhealt
hy
5Poor
Consid
er-
able
Very
hig
hV
ery
hig
hLow
to n
oStr
ong
off
ensiv
eU
nhealt
hy
10
You c
an s
ee 5
feet
or
more
deep into
the
wate
r
You c
an s
ee 2
to 5
feet
deep into
the w
ate
r
You c
an s
ee 1
to 2
feet
deep into
the w
ate
r
You c
an s
ee a
t m
ost
1 f
oot
deep into
the
wate
r
11
12
Colo
r
Blu
eBlu
e /
Bro
wn
Bro
wn/G
reen
Gre
en
A
bundant
gam
e f
ish a
nd a
few
rough f
ish
M
any g
am
e f
ish a
nd a
few
rough f
ish
M
any r
ough f
ish a
nd a
few
gam
e f
ish
A
few
rough f
ish b
ut
no g
am
e f
ish
13
14
A
lgae b
loom
s n
ever
occur
Sm
all a
mounts
of
alg
ae a
ppear
near
shore
in s
om
e
years
, and last
1 t
o 2
days
Sm
all a
mounts
of
alg
ae a
ppear
near
shore
most
years
, and last
for
about
1 w
eek
Larg
e a
mounts
of
alg
ae (
like in p
ictu
re o
n p
revio
us
scre
en)
appear
near
shore
about
once a
year
and last
for
2 t
o 3
weeks
Larg
e a
mounts
of
thic
k a
lgae a
ppear
near
shore
every
year
and last
for
most
of
the s
um
mer
15
N
o u
nple
asant
odors
1
to 2
days a
year,
fain
t odor
3
to 4
days a
year,
noti
ceable
odor
Severa
l days a
year,
noti
ceable
odor
16
Lake #
1Lake #
2
C
olo
r:Bro
wn/gre
en
C
lari
ty:C
an s
ee 1
-2
feet
deep
Fis
h:
Many r
ough f
ish
and a
few
gam
e f
ish
O
ne W
ay D
ista
nce:30
min
ute
dri
ve
C
olo
r:Blu
e/bro
wn
C
lari
ty:C
an s
ee 2
-5
feet
deep
Fis
h:
Many g
am
e f
ish
and a
few
rough f
ish
O
ne W
ay D
ista
nce:90
min
ute
dri
ve
17
Lake #
1Lake #
2
C
olo
r:Bro
wn/gre
en
C
lari
ty:C
an s
ee 1
-2
feet
deep
Fis
h:
Many r
ough f
ish
and a
few
gam
e f
ish
O
ne W
ay D
ista
nce:30
min
ute
dri
ve
C
olo
r:Blu
e/bro
wn
C
lari
ty:C
an s
ee 2
-5
feet
deep
Fis
h:
Many g
am
e f
ish
and a
few
rough f
ish
O
ne W
ay D
ista
nce:90
min
ute
dri
ve
18
Com
bin
ati
on
CC
om
bin
ati
on
B
Lake #
1Lake #
2
C
olo
r:Bro
wn/gre
en
C
lari
ty:C
an s
ee 1
-2
feet
deep
Fis
h:
Many r
ough f
ish
and a
few
gam
e f
ish
O
ne W
ay D
ista
nce:30
min
ute
dri
ve
C
olo
r:Blu
e/bro
wn
C
lari
ty:C
an s
ee 2
-5
feet
deep
Fis
h:
Many g
am
e f
ish
and a
few
rough f
ish
O
ne W
ay D
ista
nce:90
min
ute
dri
ve
19
Whic
h lake w
ould
you c
hoose?
Lake #
1Lake #
2
C
olo
r:Bro
wn/gre
en
C
lari
ty:C
an s
ee 1
-2
feet
deep
Fis
h:
Many r
ough f
ish
and a
few
gam
e f
ish
O
ne W
ay D
ista
nce:30
min
ute
dri
ve
C
olo
r:Blu
e/bro
wn
C
lari
ty:C
an s
ee 2
-5
feet
deep
Fis
h:
Many g
am
e f
ish
and a
few
rough f
ish
O
ne W
ay D
ista
nce:90
min
ute
dri
ve
20
If L
ake #
1 a
nd #
2 w
ere
your
only
opti
ons,
would
you v
isit
one o
f th
em
or
sta
y h
om
e?
21
Para
mete
rEsti
mate
Tra
vel C
ost
-0
.02
1**
Quality
Level A
1.9
88
**
Quality
Level B
1.3
83
**
Quality
Level C
0.5
73
**
Quality
Level D
-0
.20
6**
Quality
Level E
-0
.20
6**
22
Scenari
oEsti
mate
Quality
Level E =
> Q
uality
Level D
$0
.00
Quality
Level D
=>
Quality
Level C
$3
7.1
0
Quality
Level C
=>
Quality
Level B
$3
8.5
7
Quality
Level B =
> Q
uality
Level A
$2
8.8
1
Tota
l W
TP
=
Per
Tri
p W
TP
x
# o
f T
rip
s
23
Tota
l W
TP
=
Per
Tri
p W
TP
x
# o
f T
rip
s
24
Appro
xim
ate
lyA
ppro
xim
ate
ly
Tota
l W
TP
=
Per
Tri
p W
TP
x
# o
f T
rip
s
25
Appro
xim
ate
ly
What’
s m
issin
g?
Change in t
rips
Appro
xim
ate
ly
Tota
l W
TP
=
Per
Tri
p W
TP
x
# o
f T
rip
s
26
Appro
xim
ate
ly
What’
s m
issin
g?
Change in t
rips
Appro
xim
ate
ly
Leads t
o
dow
nw
ard
bia
s
Falls L
ake a
naly
sis
(von H
aefe
n, 2
010
)
Larg
e s
cale
inte
rventi
on o
ver
30 y
ears
, undis
counte
d b
enefi
ts
N
o incre
ase in t
rips: $720 M
illion
A
llow
for
incre
ase in t
rips: $1 B
illion
27
1
PU
TT
ING
IT
AL
L T
OG
ET
HE
R
Dan
Ph
aneu
f
NC
SU
2
PO
LIC
Y T
AS
K
Eval
uat
e th
e po
ten
tial
rec
reat
ion
ben
efit
s of
a pro
pose
d n
um
eric
cri
teri
a fo
r a
spec
ific
lak
e.
Exam
ple
: Jo
rdan
Lak
e, N
C
C
urr
ent
bas
elin
e as
sess
men
t is
33
g/l
chlo
rophyll
a
T
N =
0.8
2 m
g/l
, T
P =
0.0
75 m
g/l
, S
ecch
i =
0.6
0 m
, tu
rbid
ity =
12
.0 N
TU
D
efin
e a
po
licy
obje
ctiv
e of
a 15%
red
uct
ion i
n m
edia
n c
hlo
rop
hy
ll a
lev
els
to ~
28
g
/l
Wh
at
are
th
e re
crea
tion b
enef
its
of
this
poli
cy o
ver
a 2
5 y
ear
per
iod
?
3
INF
OR
MA
TIO
N N
EE
DS
1)
Wat
er q
ual
ity
ass
essm
ent
at b
asel
ine
condit
ion
s
N
eed
mo
nit
ori
ng
sta
tion r
eadin
gs
for
tota
l nit
rogen
and
ph
osp
horu
s,
chlo
roph
yll
a,
turb
idit
y,
and S
ecch
i dep
th.
2)
Co
un
terf
actu
al l
evel
s
T
arg
et v
alu
e(s)
fo
r am
bie
nt
level
s of
ind
icat
or
par
amet
er(s
)
3)
Est
imat
es o
f re
crea
tion t
rips
at b
asel
ine
condit
ion
s an
d i
n t
he
futu
re
T
ota
l an
nu
al r
ecre
atio
n t
rips
taken
to t
he
wat
er b
ody
cu
rren
tly
A
n a
ssu
mp
tio
n o
n h
ow
this
wil
l ch
ang
e due
to q
ual
ity
im
pro
vem
ents
an
d/o
r
po
pu
lati
on
gro
wth
.
4)
Tim
efra
me
and
dis
count
rate
H
ow
lo
ng
wil
l it
tak
e fo
r co
sts
and b
enef
its
to u
nfo
ld?
H
ow
sh
ou
ld w
e co
mpar
e cu
rren
t an
d f
utu
re y
ears
' ben
efit
s?
4
AS
SE
SS
ING
TH
E Q
UA
LIT
Y C
HA
NG
E
Fir
st, u
se t
he
wat
er q
ual
ity m
od
el t
o c
om
pu
te t
he
bas
elin
e in
dex
:
Rec
all
bas
elin
e fo
r Jo
rdan
Lak
e
Qua
lity
Da
ta
TN
T
P
CL
A
S
T
0.8
2 m
g/l
0
.07
5 m
g/l
3
3
g/l
0.6
0 m
1
2.0
0 N
TU
Pro
ba
bil
itie
s fo
r 5
Lev
els
Pr
1
Pr
2
Pr
4
Pr
4
Pr
5
0.0
1
0.0
4
0.3
5
0.5
5
0.0
5
Ind
ex V
alu
e
3.6
0
5
Then
, use
th
e w
ater
qu
alit
y m
od
el t
o c
om
pu
te t
he
counte
rfac
tual
in
dex
:
Co
un
terf
act
ua
l Q
ua
lity
TN
T
P
CL
A
S
T
0.7
8 m
g/l
0
.07
mg/l
2
8
g/l
0.6
0 m
1
1.2
2 N
TU
C
ou
nte
rfac
tual
val
ues
for
TN
, T
P,
S a
nd T
nee
d t
o b
e im
pu
ted
!
Co
un
terf
act
ua
l P
roba
bil
itie
s fo
r 5 L
evel
s
Pr
1
Pr
2
Pr
3
Pr
4
Pr
5
0.0
2
0.0
6
0.4
4
0.4
5
0.0
3
Ind
ex V
alu
e
3.4
2 −
no
w c
lose
r to
lev
el 3
than
lev
el 4
6
Rec
all
def
init
ion
s o
f q
uali
ty l
evel
s fo
r th
e ec
onom
ic m
od
el:
CA
TEG
OR
Y
1
2
3
4
5
CO
LOR
B
lue
B
lue/
Bro
wn
B
row
n/G
reen
B
row
n/G
reen
G
reen
CLA
RIT
Y
Can
se
e 5
fe
et d
eep
or
mo
re
Can
se
e 2
-5 f
eet
dee
p
Can
se
e 1
-2 f
eet
dee
p
Can
se
e at
mo
st 1
fo
ot
de
ep
Can
se
e at
mo
st 1
fo
ot
de
ep
FISH
A
bu
nd
ant
gam
e fi
sh a
nd
a fe
w r
ou
gh f
ish
Man
y ga
me
fish
an
d
a fe
w r
ou
gh f
ish
Man
y ro
ugh
fis
h a
nd
a fe
w g
ame
fish
A f
ew
ro
ugh
fis
h b
ut
no
gam
e fi
sh
A f
ew
ro
ugh
fis
h b
ut
no
gam
e fi
sh
ALG
AE
BLO
OM
S
Nev
er o
ccu
r Sm
all a
reas
nea
r sh
ore
;
som
e ye
ars,
1-2
day
s
Smal
l are
as n
ear
sho
re; m
ost
year
s, 1
wee
k
Larg
e ar
eas
nea
r sh
ore
;
on
ce a
yea
r, 2
-3 w
eeks
Larg
e, t
hic
k ar
eas
nea
r sh
ore
;
ever
y ye
ar, m
ost
of
sum
mer
OD
OR
N
o u
np
leas
ant
od
ors
1
-2 d
ays
a ye
ar,
fain
t o
do
r,
1
-2 d
ays
a ye
ar,
fain
t o
do
r,
3
-4 d
ays
a ye
ar,
no
tice
able
od
or,
se
vera
l day
s a
year
,
no
tice
able
od
or,
7
AS
SE
SS
ING
TH
E P
ER
-TR
IP B
EN
EF
ITS
Rec
all
fro
m R
og
er's
ta
lk:
W
TP
per
tri
p o
f m
ov
ing f
rom
lev
el 5
to l
evel
4 i
s (e
ssen
tial
ly)
$0
W
TP
per
tri
p o
f m
ov
ing f
rom
lev
el 4
to l
evel
3 i
s $
37
.10
W
TP
per
tri
p o
f m
ov
ing f
rom
lev
el 3
to l
evel
2 i
s $
38
.57
W
TP
per
tri
p o
f m
ov
ing f
rom
lev
el 2
to l
evel
1 i
s $
28
.81
We
nee
d (
wh
at a
mo
un
ts t
o)
a pro
bab
ilit
y-w
eigh
ted c
om
bin
atio
n o
f th
ese:
51
21
12
25
5(p
er-t
rip W
TP
)...
cc
cE
pp
pp
pp
Fo
r th
e Jo
rdan
Lak
e ex
ample
per
tri
p W
TP
is
$5.8
8
8
INT
ER
PR
ET
AT
ION
A t
yp
ical
vis
ito
r w
ou
ld p
ay a
n a
dd
itio
nal
$5.8
8 i
n t
ravel
-rel
ated
exp
ense
s to
vis
it
Jord
an l
ake
if t
he
aver
age
chlo
rophyll
a r
eadin
g m
ov
ed f
rom
33
g
/l t
o 2
8
g/l
.
No
tes:
T
his
do
es n
ot
mea
n s
/he
wil
l en
d u
p p
ayin
g a
n a
dd
itio
nal
$5
.88
if
ther
e is
an
imp
rov
emen
t ..
. o
nly
that
s/h
e val
ues
the
impro
vem
ent
at t
hat
lev
el.
D
oes
th
is s
eem
lar
ge
or
smal
l?
Rec
all
tha
t Jo
rda
n L
ake
wate
r qua
lity
is
a p
ub
lic
good
− a
ll v
isit
ors
ca
n e
njo
y
the
impro
vem
ent
in p
ara
llel
.
9
AN
NU
AL
AN
D T
OT
AL
PO
LIC
Y B
EN
EF
ITS
Annu
al a
gg
regat
e b
enef
its
S
ince
all
vis
itors
can
enjo
y t
he
qual
ity i
mpro
vem
ent
the
val
ue
of
the
imp
rov
emen
t in
a g
iven
yea
r is
To
tal
An
nual
Ben
efit
s =
(tr
ips
per
yea
r)×
(per
tri
p W
TP
)
Tota
l p
oli
cy b
enef
its
A
can
did
ate
for
tota
l po
licy
ben
efit
s is
(# o
f p
oli
cy y
ears
)×(t
rips
per
yea
r)×
(per
tri
p W
TP
)
B
ut
this
ig
no
res
the
dis
count
rate
...p
rese
nt
val
ue
of
flow
of
ben
efit
s is
1
(tri
ps
)(per
tri
p W
TP
)P
V B
enef
its
(1)
Tt
t
tt
rate
10
Fo
r Jo
rdan
Lak
e −
su
pp
ose
ther
e ar
e 900,0
00 t
rips
each
yea
r
A
nnu
al b
enef
its
are
(900,0
00 t
rips)
×($
5.8
8 p
er t
rip W
TP
) =
$5
.29
mil
lio
n
7
% d
isco
un
t ra
te/2
5 y
ears
pre
sent
val
ue
of
tota
l ben
efit
s =
$6
1.6
mil
lio
n
3
% d
isco
un
t ra
te/2
5 y
ears
pre
sent
val
ue
of
tota
l ben
efit
s =
$9
2.1
mil
lio
n
11
AS
SE
SS
ING
UN
CE
RT
AIN
TY
So
urc
es o
f u
nce
rtai
nty
in t
he
anal
ysi
s?
s
amp
lin
g e
rro
r in
ch
arac
teri
zati
on
of
wat
er q
ual
ity a
nd
eco
nom
ic m
od
els
a
ccu
racy
of
bas
elin
e w
ater
qual
ity a
sses
smen
t
a
ccu
racy
of
pre
dic
tion
s fo
r fu
ture
wat
er q
ual
ity c
on
dit
ion
s
e
vo
luti
on
of
wat
er q
ual
ity c
ond
itio
ns
abse
nt
inte
rven
tio
n
e
vo
luti
on
of
recr
eati
on u
se w
ith a
nd w
ithou
t po
licy
in
terv
enti
on
t
ran
sfer
abil
ity o
f ec
onom
ic a
nd w
ater
qual
ity m
od
els
bey
on
d p
rim
ary
con
tex
t
S
tuff
I a
m n
ot
thin
kin
g o
f...
12
QU
AN
TIF
YIN
G U
NC
ER
TA
INT
Y
Our
mo
del
ing
acc
ou
nts
to s
om
e ex
tent
for
firs
t so
urc
e li
sted
ab
ov
e:
Fo
r Jo
rdan
Lak
e, 3
% d
isco
unt
rate
:
mea
n =
$92
.1 m
illi
on
st
d e
rror
= 7
.9 m
illi
on
25
th %
= $
88
.2 m
illi
on
75
th %
= $
96.4
mil
lion
13
AS
SE
SS
ING
SE
NS
ITIV
ITY
Un
cert
ain
ty a
nd
dep
end
ence
on a
ssum
pti
on
s su
gges
ts a
nee
d f
or
sen
siti
vit
y
anal
ysi
s!
G
rad
ual
ly o
bta
in p
oll
uti
on r
edu
ctio
ns
over
tim
e
G
rad
ual
deg
rad
atio
n o
f w
ater
qual
ity c
ondit
ion
s ab
sen
t in
terv
enti
on
D
iffe
ren
t as
sum
pti
on
s on c
urr
ent
and f
utu
re v
isit
atio
n
N
on
-att
ain
men
t o
f poll
uti
on r
edu
ctio
n o
bje
ctiv
es
d
iffe
ren
t d
isco
un
t ra
tes
Th
e sp
rea
dsh
eet
too
l w
e'll
look
at
pro
vides
capabil
itie
s in
th
ese
dim
ensi
on
s
14
US
ING
TH
E R
ES
UL
TS
Thin
gs
to k
eep i
n m
ind
:
E
stim
ates
co
min
g o
ut
are
on
ly a
s good a
s th
e in
form
atio
n g
oin
g i
n a
nd
th
e
assu
mp
tio
ns
emp
loyed
!
S
ensi
tiv
ity
an
alysi
s is
cri
tica
l!
R
ecre
atio
n b
enef
its
are
only
one
par
t of
the
over
all
ben
efit
s
Res
ult
s ca
n p
rovi
de
a p
oin
t of
dis
cuss
ion f
or
com
pa
ring
po
licy
co
sts
an
d
ben
efit
s
Nee
d t
o a
cknow
ledg
e unce
rtain
ty
Case Study - High Rock Lake, North Carolina
A large reservoir along Yadkin River in central NC:
Listed as impaired for some time
Currently in TMDL process
319 grant in place for data collection on land use and water quality
Objective is to formulate TMDLs for chlorophyll a and turbidity
Baseline conditions:
1.27 million annual "recreation days", where a recreation day is defined as '...a visit to (the lake)
by a person for recreation purposes during a 24 hour period.' (Alcoa Recreation Use Assessment,
2004).
Monitoring station data for the months May, June, July, August, and September 2008 and 2009
are summarized as follows:
Consider the following analysis examples:
1) An ambient average target of 40 g/ml chlorophyll a over a 10 year horizon
Using the single input version of the model, what are the per trip, annual, and net present value
figures when a 7% discount rate is used?
How does the net present value change when a 3% discount rate is used?
How do things change when only baseline values for TN, TP and CLA are provided?
Redo the analysis using the multiple input version. Suppose for years 1 to 5 of the policy we
expect to achieve 42 g/ml of chlorophyll a, and that in years 6 to 10 we expect to achieve the 40
g/ml target. Suppose as well that, under the status quo, the first five baseline years are as given
by the average figures above. However, conditions are expected to deteriorate so that in years 6
to 10 the baseline conditions will be
This represents an approximate 10% decrease in measured quality. How does this affect the
estimate for total net present value at the 3% and 7% discount rates?
TN TP CLA S T
median 0.996 0.09 44 0.7 9
means 1.051648 0.108963 44.8127 0.708072 13.87044
TN TP CLA S T
1.156813 0.11986 49.29398 0.637265 15.25748
2) An ambient target of 40 g/ml chlorophyll a and 13 NTU turbidity over a 10 year horizon.
Use the single input version of the model, and compare estimates for this joint criteria to what
you obtained for chlorophyll a alone.
3) A 15% reduction in ambient concentrations of TN and TP, so that TN=0.89 mg/l and TP=0.09 mg/l.
4) Discuss the types of sensitivity analyses one might carry out for any of these scenarios. Consider in
particular:
A smaller number of recreation trips and a 5% increase in recreation trips
Benefit estimates when the target is missed - e.g. the net benefits of obtaining 42 g/ml
chlorophyll a rather than 40 g/ml.
5) What types of uncertainties are not accounted for? What types of values might be missing from these
estimates? How is an analysis like this useful (or not useful) for the types of things you work on?
Case Study - Falls Lake, North Carolina
Large reservoir on the Neuse River in NC Piedmont
A major recreation area.
Provides drinking water for nearly half a million Wake County residents.
On 303(d) list as chlorophyll a impaired.
Major reductions in TN and TP proposed (40% and 77% respectively)
A fiscal analysis for proposed nutrient strategy recently undertaken
Baseline conditions:
Approximately 1 million person visits in 2010
Measures of central tendency at current conditions
TN TP CLA S T
0.78 mg/l 0.062 mg/l 33.86 g/l 0.72 m 13.45 NTU
Measures of central tendency expected if reductions in TN and TP are achieved
TN TP CLA S T
0.71 mg/l 0.049 mg/l 23.90 g/l 0.79 m 10.39 NTU
An approximation for the baseline distribution of chlorophyll a (due to seasonal and weather
variation).
An approximation for the distribution of chlorophyll a after nutrient reductions are achieved.
24 g/l
25%
62 g/l
90%
40 g/l
75% 33 g/l
50%
24 g/l
50%
62 g/l 40 g/l
90% 33 g/l
75%
Consider the following analysis examples:
1) Suppose the criteria to evaluate is that 10% or less of chlorophyll a readings are over 40 g/l.
Given the distributions shown above, what is a sensible average chlorophyll a concentration to
evaluate?
Using the single input model and a 7% discount rate, what are the total benefits of this criteria for
a 25 year timeframe, assuming the reduction in pollution is achieved immediately?
2) It is likely that pollution reductions will occur gradually, implying the criteria level of chlorophyll a
will not occur immediately.
Consider and evaluate the 25 year benefits using the multiple input model when chlorophyll
levels decrease gradually over the first 10 years of the program. How sensitive are your results to
assumptions on when the pollution objective is obtained?
3) It is likely that, absent an intervention, conditions in Falls Lake will deteriorate from their current
level.
Consider how the 25 year benefit predictions change under different assumptions for the
downward evolution of baseline water quality.
4) It is possible that an average level of 24 g/l will not be attained in the 25 year timeframe, or that
water quality will improve beyond 24 g/l.
Consider how the 25 year benefit predictions are affected if chlorophyll a levels do not fall below
31 g/l, or if they reach 20 g/l.