test design under unobservable falsificationvirtual-md-seminar.com/slides/vsslides.pdf · timing...
TRANSCRIPT
![Page 1: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/1.jpg)
Test design under unobservable falsification
Eduardo Perez-Richet (Sciences Po) Vasiliki Skreta (UCL, UT Austin)
![Page 2: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/2.jpg)
test and faslification
tests seek to uncover some state: e.g. student’s ability; drugs potency/side effects; car’s pollution;bank’s systemic risk
decisions based on test results often by (several) third parties (‘the market’), non-coordinated,non-contractible
manipulations/ falsification /cheating, sadly, common
• standardized tests: teachers – testers – recruiters
• drugs: pharmaceuticals –FDA – (consumers)
• emissions: car manufacturers – regulator (EPA) – (consumers)
• asset rating: asset issuers – rating agencies – investors
• stress test: banks – Fed – (investors)
![Page 3: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/3.jpg)
On January 11 2017: “VW agreed to pay a criminal fine of $4.3bn for selling around 500,000 carsfitted with so-called “defeat devices" that are designed to reduce emissions of nitrogen oxide (NOx)under test conditions."
On January 12 2017: US Environmental Protection Agency (EPA) accused Fiat Chrysler Automobile ofusing illegal software in conjunction with the engines which, allowed thousand of vehicles toexceed legal limits of toxic emissions
our goal:test design in the presence of cheating
![Page 4: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/4.jpg)
Setup: Test+Falsification
![Page 5: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/5.jpg)
baseline setup
• agent: endowed with 1 or continuum of items
• Receiver(s): choose ‘pass’ or ‘fail’
• agent wants each item to be passed (payoff 1-0)• state s ∈ S ⊆ [−s, s], with −s < 0 < s , and {−s, s} ⊆ S
• S = {−s, s} as the binary state case
• items i.i.d. prior π
• prior mean µ0 ≡ Eπ(s) < 0
• Receiver(s) preferences (identical for all receivers)
• fail → 0
• pass s → s
• Receiver pass item i iff E(s) > 0
![Page 6: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/6.jpg)
timing and falsification technology
1. Test: A test τ is exogenously given and publicly observable.2. Falsification: The agent chooses a falsification strategy φ (interim same)3. State: The state s is realized according to π4. Testing and results: The falsification strategy generates a falsified state of the world s ′
according to φ, and the test generates a public signal x about the falsified state of the worldaccording to τs′
5. Receiver decision: The receiver forms beliefs and chooses to approve or reject.
Time
Tests. A test is a Blackwell experiment: a measurable space of signals X , and a Markov kernel τfrom S to ∆(X)
• π and τ define joint probability measure X × S : τπ• in the absence of falsification
• conditional on observing x , receiver forms a belief about S : τπx
• conditional on s distribution of signals depends on τ
![Page 7: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/7.jpg)
falsification technology
The agent can falsify the state of the world that is fed to the test.
• falsification φ which is a Markov kernel from S to ∆S
• if T is a Borel subset of S and s ∈ S a state of the world, then φ(T |s) denotes the probabilitythat the true state s , or source, is falsified as a target state in T
• truth-telling strategy Markov kernel δ mapping each state s to the Dirac measure δs on S
• prior π and falsification strategy φ define joint probability measure denoted φπ on S × S
• falsification costless or costly• install devices that artificially lower emission levels• teaching the students to the test• inaccurate reporting of asset characteristics• psychological lying costs
• falsification cost c(t|s) cost of falsifying source state s as target state t
• cost of falsification strategy φ is C(φ) =∫
S×Sc dφπ
![Page 8: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/8.jpg)
posterior beliefs, actions and resulting payoffs
• prior, falsification strategy and test define a joint distribution over X × S denoted by τφπ• posterior belief given x is τφπx ∈ ∆S
• µ(x |τ, φ) =∫
Ss dτφπx(s): expected state according to τφπ
• receiver approves whenever µ(x |τ, φ) ≥ 0
• signal approval set of the receiver X (τ, φ) = {x : µ(x |τ, φ) ≥ 0}
A(τ, φ) =
∫
X(τ,φ)×S
dτφπ ex ante probability of approval
U(τ, φ) = A(τ, φ) − C(φ), agent’s payoff
V (τ, φ) =
∫
X(τ,φ)×S
µ(x |τ, φ)dτφπ(x , s) receiver’s payoff
![Page 9: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/9.jpg)
agent Falsification strategyφ(s′l |s)
c(s′|s) (costs)
s′test, signal x
τ(x |s′)
post. mean µ(x |τ, φ)
action
a(µ) =
{
1 if µ(x|τ, φ) > 0
0 otherwise
unobservable (no commitment) receiver acts given x
test public
![Page 10: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/10.jpg)
agent Falsification strategyφ(s′l |s)
c(s′|s) (costs)
s′test, signal x
τ(x |s′)
post. mean µ(x |τ, φ)
action
a(µ) =
{
1 if µ(x|τ, φ) > 0
0 otherwise
unobservable (no commitment) receiver acts given x
test publicobservable (commitment) in paper
![Page 11: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/11.jpg)
Committed versus non-committed falsification
beliefs with observable (committed) falsification the meaning of x ‘reacts’ to actual φ
beliefs with unobservable (non-committed) falsification meaning depends on τ ; equilibriumfalsification φE
• with commitment agent is a “constrained" persuader: instead of choosing any experiment, hecan only induce information structures consistent with τ
• signals 6= action recommendations• → need continuum “pass" signals even binary state• challenge 2: entire information structure & approval thresholds change with falsification
![Page 12: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/12.jpg)
Committed versus non-committed falsification
beliefs with observable (committed) falsification the meaning of x ‘reacts’ to actual φ
beliefs with unobservable (non-committed) falsification meaning depends on τ ; equilibriumfalsification φE
• with commitment agent is a “constrained" persuader: instead of choosing any experiment, hecan only induce information structures consistent with τ
• signals 6= action recommendations• → need continuum “pass" signals even binary state• challenge 2: entire information structure & approval thresholds change with falsification• in Perez-Richet and Skreta (2018)
![Page 13: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/13.jpg)
Committed versus non-committed falsification
beliefs with observable (committed) falsification the meaning of x ‘reacts’ to actual φ
beliefs with unobservable (non-committed) falsification meaning depends on τ ; equilibriumfalsification φE
• with commitment agent is a “constrained" persuader: instead of choosing any experiment, hecan only induce information structures consistent with τ
• signals 6= action recommendations• → need continuum “pass" signals even binary state• challenge 2: entire information structure & approval thresholds change with falsification• in Perez-Richet and Skreta (2018)
![Page 14: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/14.jpg)
Committed versus non-committed falsification
beliefs with observable (committed) falsification the meaning of x ‘reacts’ to actual φ
beliefs with unobservable (non-committed) falsification meaning depends on τ ; equilibriumfalsification φE
• with commitment agent is a “constrained" persuader: instead of choosing any experiment, hecan only induce information structures consistent with τ
• signals 6= action recommendations• → need continuum “pass" signals even binary state• challenge 2: entire information structure & approval thresholds change with falsification• in Perez-Richet and Skreta (2018)
• NEW unobservable falsification• akin to mechanism design without transfers• here signals =action recommendations• if falsification costless: WLOG no falsification (a.k.a “truth-telling") best response• but without costs no test works....• characterisation of optimal test: involves falsification!• derivation of falsification proof test
![Page 15: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/15.jpg)
overview of results
• general framework to study manipulations• mechanism design with costly reports; no transfers• issues with revelation principle• optimum involves lying–and lying is essential• optimal falsification-proof test strictly worse
• constrained infinite dimensional program• usual relaxed program not ususefull• non-local IC bind• and continuum of binding IC• novel characterization via auxiliary problem/dual of optimal transportation problem
![Page 16: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/16.jpg)
Warm-up Binary state
![Page 17: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/17.jpg)
baseline setup
• agent: endowed with 1 or continuum of items
• agent wants each item to be passed (payoff 1-0)• each S = {−s , −s}
• distributed i.i.d. with Pr(s = s) = π0;
• Receiver(s) preferences (identical for all receivers)
• fail → 0
• pass s → s > 0
• pass − s → − s < 0
• Receiver pass item i iff Pr(s = s) ≥ 0,• test τ and τ• falsification state −s generates signals from τ : φ( WLOG ignore ‘downwards’ falsification)
![Page 18: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/18.jpg)
fully informative test receiver-optimal without cheating
s
−s
x
x
π
1−π
1
1
0
PA
SS
FA
IL
PAYOFFS
Receiver:∅ πs
agent:∅ π
![Page 19: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/19.jpg)
agent-optimal a.k.a. Kamenica-Gentzkow test
s
−s
x
x
0
PA
SS
FA
IL
π
1−π
1
1− π0 s(1−π0)s
π0s
(1−π0)s
PAYOFFS
Receiver:∅KG
FI
agent:∅ FI KG
![Page 20: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/20.jpg)
falsification of two-signal tests
Suppose there is a fully revealing two-signal test: X = {x , x}
• suppose φ observable–endogenously costly “devalues” signals• signal x yields pass if: π0s − (1 − π0)φs > 0
• π0 + φ(1 − π0)(1 − c)1
(
φ ≤ π0s
(1−π0)s
)
• if 1 − c > 0 optimal φ is φ = π0s
(1−π0)s
• setting φ = π0s
(1−π0)sand “approve" after x is an equilibrium if φ observable
• agent achieves optimum!
• no equilibrium with pos. prob of “approve" if φ unobservable (if that was the case agentchooses φ = 1, but then receiver never approves
• both benefit when falsification observable/detectable• this talk: what can be done in unobservable case when falsification is costly
![Page 21: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/21.jpg)
fully informative test + observable falsification
s
−s
s
−s
Signalx
x
Expectations
−s
Falsified StateState
0
PA
SS
FA
IL
π
1−π
1
1− π0s
(1−π0)s
π 0s
(1−
π 0)s
1
1
PAYOFFS
Receiver:∅KG
f ◦FI
FI
agent:∅ FI KG
f ◦FI
![Page 22: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/22.jpg)
Undetectable falsification: two states
Binary State: Set of feasible tests given by:
• WLOG test as an approval probability τ, τ
• “Falsification proofness"/informativeness condition: τ − τ ≤ c
• otherwise −s falsifies as s ; → no information
• Obedience Constraint: τπ0s − τ(1 − π0)s ≥ 0
![Page 23: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/23.jpg)
Binary State
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
�
�
c = 0
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
�
�
c � 1� �0s/ (1� �0)s
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
c � 1� 0s/ (1� �0)s
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
�
�
c = 1
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Receiver’s Payof
Agent’sPayof
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Receiver’s Payof
Agent’sPayof
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
�0
�R
�P
�A
Receiver’s Payof
Agent’sPayof
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
�0
�R
�A
Receiver’s Payof
Agent’sPayof
![Page 24: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/24.jpg)
Undetectable falsification: general state space
![Page 25: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/25.jpg)
receiver optimal test: program formulation–preliminaries
signals: action recommendations
Let φ be an equilibrium falsification strategy under τ . Then the test τ ′ with binary signalspace X ′ = {Pass, Fail} defined by
τ ′(Pass|s) = τ (X(
τ, φ)|s)
is such that φ is an equilibrium under τ ′ is equivalent in terms of payoffs and approvals. So:
• we redefine tests as measurable functions τ : S → [0, 1]
• nominal passing probability τ (s): probability test recommends passing s
• falsification induces the “ true" passing probability that can differ from nominal
![Page 26: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/26.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience
![Page 27: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/27.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience
![Page 28: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/28.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience
![Page 29: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/29.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience
![Page 30: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/30.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience redundant
![Page 31: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/31.jpg)
supτ,φ
∫
S×S
sτ (t)dφπ(t, s)
s.t.∫
S×S
{
τ (t) − c(t|s)}
dφπ(t, s) ≥
∫
S×S
{
τ (t) − c(t|s)}
dφ′π(t, s), ∀φ′ ex-ante optimal falsification∫
S×S
sτ (t)dφπ(t, s) ≥ 0 receiver obedience redundant
ex ante optimal falsification (EOF) is equivalent interim (IOF): φs puts probability 1 on set of(interim) optimal falsification targets
program
supτ,φ
∫
S×S
sτ (t)dφπ(t, s) (P)
s.t. φ (Φ(s; τ )|s) = 1, ∀s ∈ S (IOF)where Φ(s; τ ) = argmaxt τ (t) − c(t|s) (optimal falsification targets)
![Page 32: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/32.jpg)
assumptions on falsifications costs
cost: given c(t|s) where c : S × S → R+ is measurable, and continuous in t .
• c(s|s) = 0.• Monotonicity (MON): If c 6= 0, c(t|s) increasing in t and decreasing in s if s < t , ...• Triangular inequality (TRI): c(t|m) + c(m|s) ≥ c(t|s).
Lets0 = max
{
s′ ∈ S : Eπ(s|s ≥ s
′) ≤ 0}
.
In particular, if π has no atom at s0, then Eπ(s|s ≥ s0) = 0.
![Page 33: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/33.jpg)
further simplifications
• if the cost function satisfies (TRI), we can restrict attention to tests that arefalsification proof for negative states
• we can also restrict attention to tests when positive states don’t have incentive tofalsify as negative
![Page 34: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/34.jpg)
optimal class of tests
We now consider a class of tests defined by two parameters: the highest nominal passingprobability p ∈ [0, 1], and the cutoff state s ∈ S+ above which nominal probabilities are set to p:
τp,s(s) =
{
p for s ≥ s[
p − c(s |s)]+ for s < s
here s(p, s) = min {s ≤ s : c(s |s) ≤ p} and such that s(p, s) ≤ 0.
properties
(i) τp,s is continuous on S , strictly increasing on(
s, s]
and constant and equal to 0 belows(p, s) and and constant and equal to p above s
(ii) if the cost function satisfies (TRI), then truth-telling optimal for every s ∈ S
(s ∈ Φ(s; τp,s )).(iii) for every s ∈
(
s(p, s), s]
, falsify to s ALSO optimal s ∈ Φ(s; τp,s)
(iv) receiver-preferred falsification s ∈(
0, s)
falsify to s
![Page 35: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/35.jpg)
optimal test
Theorem
Suppose the cost function satisfies (TRI). Then (τp∗,s∗ , φp∗,s∗ ) maximizes (P), where
p∗ = min
{
c(s|s0), 1}
,
ands
∗ = max{
s ∈ S : c(s|0) ≤ 1}
.
Furthermore, the receiver gets her first-best payoff if and only if c(s|0) ≥ 1. However, the pair ofresulting payoffs (U∗,V ∗) never lies on the Pareto frontier.
![Page 36: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/36.jpg)
State
τ (s)
-10 -5 0 50
0.5
1 Optimal test (nominal)Optimal test (induced)
(a) c(t|s) = α(t − s), α = 115
State
τ (s)
-10 -5 0 50
0.5
1
(b) c(t|s) = α√
t − s , α = 13
State
τ (s)
-10 -5 0 50
0.5
1
(c)c(t|s) = αe2βs
{
(t − s) + β(t − s)2}
,
α = 120
and β = 130
State
τ (s)
-10 -5 0 50
0.5
1
(d) c(t|s) = α
{
(t − s) − β(t − s)2}
,
α = 110
and β = 150
![Page 37: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/37.jpg)
Proof
Step 1:
• use (TRI) to show that we can transform any test so that negative states are truthful whileimproving both receiver and agent payoffs.
• IDEA: give negative states their falsification payoff
Step 2: we can further transform any test so that nonnegative states do not falsify as negativestates while improving both receiver and agent payoffs.
Step 3: we can replace any such transformed test by a test of the form τp,s and increase thereceiver’s payoff.
Step 4: optimize on p and s
![Page 38: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/38.jpg)
optimal falsification proof test
![Page 39: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/39.jpg)
supτ
∫
S
sτ (s)dFπ(s) (FPProg)
s.t. τ (t) − τ (s) ≤ c(t|s), ∀s, t ∈ S (FPIC)
properties of FPIC tests
Let τ be a test that satisfies (FPIC) then:1. τ is continuous2. there exists a K-Lipschitz and nondecreasing test function τ that also satisfies (FPIC)and makes the receiver better off
3. =⇒ test increasing and K-Lipschitz, differentiable a.e. derivative τ ′ bounded in [0,K ]
These properties result to envelope characterization τ (s) = τ +∫ s
−sτ ′(z)dz
Can choose scaler τ ∈ [0, 1] and the function {τ ′(s)}s∈S
Need: Regularity (REG): c(t|s) is continuously differentiable in t on [s, s] and in s on [−s, t],and there exists K > 0 such that, for every t > s ,
c(t|s) ≤ K(t − s).
![Page 40: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/40.jpg)
an auxiliary function
Let J : S → R be:
J(z) =
∫ s
z
sdFπ(s)
properties of J(z)
1. J(z) < 0 for z < s0
2. J(z) ≥ 0 for z ≥ s0
3. continuous4. increasing on S−
5. decreasing and S+
6. =⇒ single-peaked at 0 − 3 − 2 − 1 0 1 2
− 0.4
− 0.2
0.0
0.2
0.4 J()
λ
π=Uniform([−3,2])
![Page 41: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/41.jpg)
reformulating the program
τµ0 +
∫
S
τ ′(z)J(z)dz (reformulated objective function of (FPProg))
s.t.τ +
∫
S
τ ′(z)dz ≤ 1 (probability bound)∫ t
s
τ ′(z)dz ≤ c(t|s), for all s < t (FPIC)
Reducing τ increases the objective function as µ0 < 0, relaxes the probability constraint, and hasno effect on the incentive constraints, implying that it is optimal to set τ = 0
![Page 42: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/42.jpg)
solving a relaxed program
We ignore FPIC and solve relaxed program, where we treat the probability constraint with theLagrangian method. Let λ ≥ 0
L(τ ′, λ) =
∫
S
τ ′(z)J(z)dz + λ
(
1 −
∫
S
τ ′(z)dz
)
=
∫
S
τ ′(z)(
J(z) − λ)
dz + λ
Maximize L(τ ′, λ) where τ ′ : S → [0,K ] is feasible if, for every s < t ,∫ t
sτ ′(z)dz ≤ c(t|s), and
∫ s
s0τ ′(z)dz ≤ 1.
Any solution must satisfy τ ′(s) = 0 for almost every s such that J(s) < λ, that is, by continuity andsingle-peakedness of J , outside of an interval [s∗, s
∗] such that J(s∗) = J(s∗) = λ.
![Page 43: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/43.jpg)
Lemma (Lagrangian sufficiency theorem)
Suppose that there exists λ ≥ 0, and a feasible τ ′ such that:
(a) λ = 0 or∫
Sτ ′(z)dz = 1;
(b) For every feasible τ ′, L(τ ′, λ) ≥ L(τ ′, λ).
Then there exists an interval [s∗, s∗] such that:
(i) s0 ≤ s∗ ≤ 0 ≤ s∗ ≤ s and J(s∗) = J(s∗) = λ;(ii) τ ′(s) = 0 for every s /∈ [s∗, s
∗];
(iii) The test τ(s) =(
∫ s∗∧s
s∗τ ′(z)dz
)
1(s ≥ s∗) is a falsification-proof receiver optimal test.
![Page 44: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/44.jpg)
matching function
Matching function help us guess optimal Lagrange multiplier: Choose
s∗ = min{s ∈ [s0, 0] : c(m(s)|s) ≤ 1},
Then: λ∗ = J(s∗) Note that s∗ = s0 whenever c(s |s0) ≤ 1
• matching function m : [s0, 0] → [0, s] is• decreasing• implicitly defined by J(s∗) = J(m(s∗))
• or equivalently by∫ m(s∗)
s∗sdFπ(s) = 0
• s0 is matched with m(s0) = s
• each choice of s∗ ∈ [s0, 0] uniquely pins down s∗ = m(s∗)
![Page 45: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/45.jpg)
next step: program reformulation
Instead of solving the Lagrangian problem, we go back to the original program. Focus on tests τthat are constant outside of [s∗, s
∗], and τ (s∗) = 0.
Also relax the program by getting rid of the constraint that τ (s∗) ≤ 1, and only keeping theincentive constraints for pairs (s, t) such that s∗ ≤ s ≤ 0 ≤ t < s∗
Change variables and let y = −s ∈ Y = [0,−s∗] and z = t ∈ Z = [0, s∗]. Finally, we let ρ : Y → R,and ψ : Z → R be the functions defined by ρ(y) = τ (−y) = τ (s), and ψ(z) = τ (z) = τ (t). Withthese notations, the remaining incentive constraints become
ψ(z) − ρ(y) ≤ c(z| − y), ∀(y , z) ∈ Y × Z .
And, up to multiplication by the constant µ∗ =∫ s∗
0sdFπ(s), the objective function of the program
becomes∫
Z
ψ(z)dQ(z) −
∫
Y
ρ(y)dP(y),
where Q(z) = 1µ∗
∫ z
0xdFπ(x), and P(y) = 1
µ∗
∫ y
0xdFπ(−x) define atomless cumulative distribution
functions on, respectively, Z and Y .
![Page 46: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/46.jpg)
new relaxed and reformulated program:
supρ,ψ
∫
Z
ψ(z)dQ(z) −
∫
Y
ρ(y)dP(y)
s.t. ψ(z) − ρ(y) ≤ c(z| − y), ∀(y , z) ∈ Y × Z ,
is dual of the following well known Monge-Kantorovich optimal transport problem
infϕ∈M(P,Q)
∫
Z×Y
c(z| − y)dϕ(z, y),
where M(P,Q) is the set of joint distributions on Z × Y with marginals Q on Z , and P on Y .
Assume: Upward increasing differences (UID): c(t ′|s ′) − c(t|s ′) ≥ c(t ′|s) − c(t|s) for s < s ′ ≤ t < t ′
=⇒ transportation cost function of this problem, c(z| − y) is submodular, well known solutionfor both problems
![Page 47: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/47.jpg)
Let ct and cs we denote the partial derivatives of the cost function.
τ∗(s) =
−∫ s
s∗cs
(
m(x)|x)
dx for s ∈ [s∗, 0]
c(s∗|s∗) −∫ s∗
sct
(
x |m−1(x)dx)
for s ∈ (0, s∗]
1 for s > s∗
The following theorem shows that τ∗ solves our initial problem.
Theorem
The test τ∗ solves (FPProg) and is therefore a receiver-optimal falsification-proof test. Thecorresponding receiver’s payoff is given by
U(τ∗, δ) =
∫ 0
s∗
−sc(
m(s)|s)
dFπ(s) =
∫ s∗
0
tc(
t|m−1(t))
dFπ(t).
Furthermore, the outcome (τ∗, δ) is Pareto inefficient.
![Page 48: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/48.jpg)
State
τ (s)
-10 -5 0 50
0.5
1 Falsification proof test
(e) c(t|s) = α(t − s), α = 115
State
τ (s)
-10 -5 0 50
0.5
1
(f) c(t|s) = α√
t − s α = 13
State
τ (s)
-10 -5 0 50
0.5
1
(g)c(t|s) = αe2βs
{
(t − s) + β(t − s)2}
,
α = 120
and β = 130
State
τ (s)
-10 -5 0 50
0.5
1
(h) c(t|s) = α
{
(t − s) − β(t − s)2}
,
α = 110
and β = 150
![Page 49: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/49.jpg)
State
τ (s)
-10 -5 0 50
0.5
1 Falsification proof testOptimal test (nominal)Optimal test (induced)
(i) c(t|s) = α(t − s), α = 115
State
τ (s)
-10 -5 0 50
0.5
1
(j) c(t|s) = α√
t − s , α = 13
State
τ (s)
-10 -5 0 50
0.5
1
(k)c(t|s) = αe2βs
{
(t − s) + β(t − s)2}
,
α = 120
and β = 130
State
τ (s)
-10 -5 0 50
0.5
1
(l) c(t|s) = α
{
(t − s) − β(t − s)2}
,
α = 110
and β = 150
![Page 50: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/50.jpg)
Payoff plots
0.0 0�1 0�2 0�3 0�4
0�0
0�2
0�4
0 6
0.8
Receiver
Agent
R class
R-opt imal
FPPE class
FP-R-opt imal
c(t|s)=1.33|t−s|
1+|t−s|π=Uniform([−3,2])
![Page 51: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/51.jpg)
literature
information design / Bayesian persuasion:
• Kamenica and Gentzkow (2011), Gentzkow and Kamenica (2016), Kolotilin (2016)• with manipulations: Frankel and Kartik (2019), Guo and Shmaya (2019), Nguyen and Tan (2020)
mechanism design with costly reporting:
• Kephart and Conitzer (2016), Deneckere and Severinov (2017), Severinov and Tam (2019)
mechanism design without transfers:
• Amador and Bagwell (2013), Amadon, Werning, Angeletos (2006), Ben-Porath, Dekel andLipman (2014)
costly state falsification:
• mechanism design: Lacker and Weinberg (1989), Landier and Plantin (2016)• testing: Cunningham and Moreno de Barreda (2015)
![Page 52: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/52.jpg)
Observable falsification in Perez-Richet and Skreta (2018)
![Page 53: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/53.jpg)
test + observable falsification: a 3-signal test
State
s
−s
Falsified State
s
−s
Signalx
x
Expectations
s
0
AP
PR
OV
ER
EJE
CT
π0
1−π0
1
1−φ
φ
o
1−τ
τ
τ
1−τ
Eτφπ(s|x)
Eτφπ(s|o)
Eτφπ(s|x)
PAYOFFS
Receiver:∅KG
f ◦FI
FI
agent:∅ FI KG
f ◦FI
![Page 54: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/54.jpg)
test + falsification: a 3-signal test for observable case
s
−s
G
B
Signal1
0
Belief1
0
Falsified StateState
0
PA
SS
FA
IL
π
1−π
1
1
0
11−0
1−02−0
1− π(1−0)2(1−π)0(2−0)
π(1−0)2
(1−π)0(2−0)
PAYOFFS
Receiver:∅KG
f ◦FI
FI
agent:∅ FI KG
f ◦FIf ◦3S
f ◦3S
![Page 55: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/55.jpg)
second result (observation)
adding an extra (noisy) signal helps!
the 3-signal test contains a simple practical insight: introducing a “noisy" (pooling) gradethat is associated with approval in the absence of falsification, can make falsification socostly that it prevents it, rendering this noisy test much better than the (manipulated) fullyinformative test
next
![Page 56: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/56.jpg)
second result (observation)
adding an extra (noisy) signal helps!
the 3-signal test contains a simple practical insight: introducing a “noisy" (pooling) gradethat is associated with approval in the absence of falsification, can make falsification socostly that it prevents it, rendering this noisy test much better than the (manipulated) fullyinformative test
next
• is the three signal test optimal?• how many signals do we need?• is optimal test falsification-proof?• how can we tractably find it?
![Page 57: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/57.jpg)
receiver-optimal test with observable falsification
results in a nutshell
• any test feasible with unobservable fals, feasible with observable• with 2 states/ establish falsification proofness–like “revelation principle"
• intuition: test + optimal cheating = new test → offer new test• no incentive to cheat in new test–otherwise cheating not optimal in old test• argument can fail with certain costs/more than two states
• formulate tractable program derive optimum• optimal test is rich: signals 6= recommendations
• one failing signal• a continuum of passing signals• clustering of signals above the approval threshold• good type only generates “pass” signals• bad type may generate both “pass” or “fail” signals• payoffs on Pareto Frontier• makes agent indifferent across all falsification levels (thresholds)
![Page 58: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/58.jpg)
concluding remarks
Both the receiver and the agent STRICTLY benefit if falsification is observable
It is useful to publish the empirical distribution of grades: simple yet important practicalinsight: tests can gain credibility if the principal publishes the empirical distribution of testresults. doing so enables fraud detection
Test credibility benefits everyone ex-ante
![Page 59: Test design under unobservable falsificationvirtual-md-seminar.com/slides/VSSlides.pdf · timing and falsification technology 1. Test: A test τis exogenously given and publicly](https://reader033.vdocuments.site/reader033/viewer/2022050201/5f547f125c3c32588a56616b/html5/thumbnails/59.jpg)
thank you!