read montague baylor college of medicine houston, tx reward processing and social exchange
TRANSCRIPT
Read MontagueBaylor College of Medicine
Houston, TXwww.hnl.bcm.tmc.edu
Reward Processing and Social Exchange
I love you
Let’s do lunch
BA
A BAB
Natural visual statistics
Specific algorithms evolved to solve efficiently the array of problems of early
vision
Recharge or die
The real world imposes another difficult requirement on all mobile organisms
Natural reward-harvesting statistics?
genericwoodlandcreature
Models of reinforcement learning can provide insight
What computations should we expect?
Reward harvesting is an economic problem involving both valuation and choice
Sustain goal
Select goal
Pursue goal
Goal-directed choice
Actual experience Counterfactual experience‘what could have been’
guidancesignals
Midbrain dopamine neurons
burst
pause
R timenaive
R timeAfter learning
timeAfter learning(catch trial)
Pause, burst, and ‘no change’ responses represent reward prediction errors – ongoing emission of information
ERROR SIGNAL = current reward + next prediction - current prediction
Can we detect a reward prediction error signal in a human subject?
Neural correlates of these error signals have been observed in conditioning and decision
tasks
Passive conditioning tasks
Instrumental conditioning tasks
Sequential decision tasks
Social and economic exchange tasks
Midbrain dopamine neurons
burst and pause responses encode reward prediction errors
ERROR SIGNAL = current reward + next prediction - current prediction
Can these systems be re-deployed for abstractly defined rewards (ideas)?
“I want to solve Fermat’s last theorem”
What about the something more abstract like the expression and
repayment of trust?
Trust
Modeling
Must involve risk (uncertainty)
Harvesting returns from another agent
TRUST
I love you
Let’s do lunch
BA
A BAB
Simplifying and quantifying Trust(Berg et al., 1995; Weigelt and Camerer, 1988)
Trust is the amount of money a sender sends to a receiver without external enforcement.
Agent 2Agent 1
I
R
X 3
Investor Trustee
$20
A dynamic version of the Trust game (10 rounds)
investperiod
8 s 8 s10 s 10 s 10 s8 s8 s4 s4 s
cue to invest
repaymentrevealed toboth brains
Gave
13
Kept
29
investmentrevealed toboth brains
Kept Gave
6 14
repayperiod
cue to repay
delayperiod
delayperiod
delayperiod
delayperiod
investment phase repayment phase
Totals
2919
totalsrevealed toboth brains
inter-rounddelay period
investperiod
8 s 8 s10 s 10 s 10 s8 s8 s4 s4 s
cue to invest
repaymentrevealed toboth brains
Gave
13
Kept
29
Gave
13
Kept
29
investmentrevealed toboth brains
Kept Gave
6 14
Kept Gave
6 14
repayperiod
cue to repay
delayperiod
delayperiod
delayperiod
delayperiod
investment phase repayment phase
Totals
2919
Totals
2919
totalsrevealed toboth brains
inter-rounddelay period
Structure of a round
What is the behavioral signal that most strongly
influences changes in trust (money sent) ?
Agent 2Agent 1
I
R
Reciprocity = TIT-FOR-TAT
Reciprocity = TIT-FOR-TAT
mon
ey s
en
tto
part
ner
+Benevolent
signal
-Malevolent
signal
Neutralsignal
Questions about brain response
Responses that differentiate benevolent from neutral?
Responses that differentiate malevolent from neutral?
Responses that differentiate benevolent from malevolent?
Surprise response
*
time (sec)-8 0 10
-0.2
0
0.2
Trustee Brain:
‘intention to increase trust’ shifts with reputation building
Reputation develops
Trustee will increase trust on next move
Trustee will decreasetrust on next move
submit reveal*
increases or decreasesin future trust by trustee
-0.2
0
0.2
time (sec)
Signal is nowanticipating the
outcome
Temporal shift resembles value transfer in reward learning experiments
burst
pause
R timenaive
R timeAfter learning
timeAfter learning(catch trial)
-1
-.9
-.8
-.7
-.6
-.5
-.4
-.3
-.2
-.1
0
-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1
r = .00TR
US
TE
E
decr
ease
s re
paym
en
t
Trustee decreases No information
0
.1
.2
.3
.4
.5
.6
.7
.8
.9
1
-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1
r = .27
Change in next investment
TR
US
TE
E in
crease
s re
paym
en
t
Trustee decreases positive correlation
Why should intentions to increase repayment burn more energy?
Change in next investment
1. Social trust is about modeling others - always some underlying currency.
2. Re-deploy reward-harvesting machinery that we share with all vertebrates (abstractions gain reward status)
3. Use to probe pathologies (addicted state, autism spectrum disorder, borderline…)
Trust?
Baylor College of MedicinePearl Chiu
Amin KayaliBrooks King-Casas
Terry LohrenzSam McClure (Princeton)
Read MontagueDamon Tomlin
CaltechCedric Anen
Colin CamererSteve Quartz
UCLPeter Dayan
Nathaniel Daw
Salk InstituteTerry Sejnowski
Princeton UniversityJon Cohen www.hnl.bcm.tmc.edu/trust
Emory UniversityGreg Berns
University of AlabamaLaura KlingerMark Klinger
Families of autistic subjects
Funding SourcesNIDANIMH
Dana FoundationKane Family Foundation
Angel Williamson Imaging CenterInstitute for Advanced Study, Princeton NJ
Collaborators
rounds
.1
.2
.3
.4
3 4 5 6 7 8 9 10
fractionof
highlyaccurateguesses
by trustee
Investorevents
Trusteeevents
How do we know a reputation is forming?
8 s 10 s4 s. . .
investperiod
cue to invest
investmentrevealed toboth brains
delayperiod
guessperiod
cue to guess
Investment phase
synchronized onsubmission time of both
partners
Trustee
What is Fair?
i
i
i
Split the profit
Split ‘loaf’Split the total 10 + i each
i
Investor Trustee
20-i
collective ownership? common goods?