john duchi - stanford universityjduchi/projects/dawn-duchi.pdf · msr learning to rank [qin &...

39
Uniformity in Learning John Duchi (Subbing for Hong Namkoong, Tatsu Hashimoto, Feng Ruan)

Upload: others

Post on 23-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Uniformity in LearningJohn Duchi

(Subbing for Hong Namkoong, Tatsu Hashimoto, Feng Ruan)

Page 2: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: the classical view

nData set sizeStat

istic

al P

erfo

rman

ce

Page 3: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

ce

nData set size Other

resources

Page 4: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

ce

nData set size

Computation

Page 5: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

ce

nData set size Energy u

se

Page 6: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

ce

nData set size Disc

losure

risk (p

rivacy)

Page 7: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

ce

nData set size Fairness

or

Robustness

Page 8: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Statistical learning: our view

Stat

istic

al

Perfo

rman

cen

Data set size Other

resources

Fundamental limits bound performance

Algorithms to improve performance

Machine learning, optimization, information

theory, statistics

Page 9: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Challenges in Statistical Learning• Machine learning models are increasingly used in

high-stakes decisions

- Diagnosing diseases, prescribing hiring decisions, and evaluating loan applications

• Reliable performance is crucial

- Models need to work uniformly well on heterogeneous subpopulations

• Standard methods perform poorly on subpopulations

Algorithms now rank job candidates at LinkedIn, and match candidates to division at Goldman Sachs

Page 10: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• Warfarin is the most widely used blood-thinner worldwide

• Task: learn to predict therapeutic warfarin dosage

• Personalized treatment recommendation based on regression models [International Warfarin Pharmacogenetics Consortium ’09]

- Worked best out of polynomial regression, kernel methods, neural networks, regression splines, boosting [IWPC ’09]

Example: Predicting Warfarin Dosage

Tail performance is orders of magnitude worse than average

Another use for Warfarin: rat poison

Page 11: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Challenge 1: Long-tails• Long-tailed data is ubiquitous in modern applications

- At Google, large fraction of queries are new each day

• Tail inputs often determine quality of service

queries

num

ber o

f exa

mpl

es

Long-tailed queries

MSR Learning to Rank [Qin & Liu ’13]

Beat

les

Hong

and

Tats

u

John

Page 12: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Other examples:

• Language identification [Blodgett et al. ’16, Jurgens et al. ’17]

• Part of speech tagging [Hovy & Sgaard ’15]

• Video captioning [Tatman ’17]

• Recommenders [Ekstrand et al. ’17, ’18]

Challenge 2: Lack of Diversity in Data

• “Clinical trials for new drugs skew heavily white”- From 1993-2013, 98.1% of all studies on respiratory diseases

did not report inclusion of minority subjects [Burchard et al. ’18]

- Racial minorities more likely to suffer from respiratory diseases

• Majority of image data from US & Western Europe

ImageNet: country of origin

[Shankar et al. ’17]

[Oh et al. ’15, Burchard et al. ’15, SA Editors ’18]

Page 13: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

The DAWN stack

Democratizing machine learning is not just for developers

Page 14: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• Loss/Objective where is parameter/decision to be learned, and is random data

• Optimize average performance under

Linear regression

SVM (Classification)

Deep neural networks

Standard Approach: Average Loss

`(✓;Z)<latexit sha1_base64="G4bgTcRNqAe+3O9MC0QuroZMCoA=">AAAB+HicbZDLSgNBEEV7fMb4SNSlm8YgxE2YEUHBTdCNywjmgZkh9HRqkiY9D7prhDjkS9y4UMStn+LOv7GTzEITLzQcblVR1ddPpNBo29/Wyura+sZmYau4vbO7VyrvH7R0nCoOTR7LWHV8pkGKCJooUEInUcBCX0LbH91M6+1HUFrE0T2OE/BCNohEIDhDY/XKJRekrLo4BGRX9OG0V67YNXsmugxODhWSq9Erf7n9mKchRMgl07rr2Al6GVMouIRJ0U01JIyP2AC6BiMWgvay2eETemKcPg1iZV6EdOb+nshYqPU49E1nyHCoF2tT879aN8Xg0stElKQIEZ8vClJJMabTFGhfKOAoxwYYV8LcSvmQKcbRZFU0ITiLX16G1lnNMXx3Xqlf53EUyBE5JlXikAtSJ7ekQZqEk5Q8k1fyZj1ZL9a79TFvXbHymUPyR9bnD4IqklI=</latexit><latexit sha1_base64="G4bgTcRNqAe+3O9MC0QuroZMCoA=">AAAB+HicbZDLSgNBEEV7fMb4SNSlm8YgxE2YEUHBTdCNywjmgZkh9HRqkiY9D7prhDjkS9y4UMStn+LOv7GTzEITLzQcblVR1ddPpNBo29/Wyura+sZmYau4vbO7VyrvH7R0nCoOTR7LWHV8pkGKCJooUEInUcBCX0LbH91M6+1HUFrE0T2OE/BCNohEIDhDY/XKJRekrLo4BGRX9OG0V67YNXsmugxODhWSq9Erf7n9mKchRMgl07rr2Al6GVMouIRJ0U01JIyP2AC6BiMWgvay2eETemKcPg1iZV6EdOb+nshYqPU49E1nyHCoF2tT879aN8Xg0stElKQIEZ8vClJJMabTFGhfKOAoxwYYV8LcSvmQKcbRZFU0ITiLX16G1lnNMXx3Xqlf53EUyBE5JlXikAtSJ7ekQZqEk5Q8k1fyZj1ZL9a79TFvXbHymUPyR9bnD4IqklI=</latexit><latexit sha1_base64="G4bgTcRNqAe+3O9MC0QuroZMCoA=">AAAB+HicbZDLSgNBEEV7fMb4SNSlm8YgxE2YEUHBTdCNywjmgZkh9HRqkiY9D7prhDjkS9y4UMStn+LOv7GTzEITLzQcblVR1ddPpNBo29/Wyura+sZmYau4vbO7VyrvH7R0nCoOTR7LWHV8pkGKCJooUEInUcBCX0LbH91M6+1HUFrE0T2OE/BCNohEIDhDY/XKJRekrLo4BGRX9OG0V67YNXsmugxODhWSq9Erf7n9mKchRMgl07rr2Al6GVMouIRJ0U01JIyP2AC6BiMWgvay2eETemKcPg1iZV6EdOb+nshYqPU49E1nyHCoF2tT879aN8Xg0stElKQIEZ8vClJJMabTFGhfKOAoxwYYV8LcSvmQKcbRZFU0ITiLX16G1lnNMXx3Xqlf53EUyBE5JlXikAtSJ7ekQZqEk5Q8k1fyZj1ZL9a79TFvXbHymUPyR9bnD4IqklI=</latexit><latexit sha1_base64="G4bgTcRNqAe+3O9MC0QuroZMCoA=">AAAB+HicbZDLSgNBEEV7fMb4SNSlm8YgxE2YEUHBTdCNywjmgZkh9HRqkiY9D7prhDjkS9y4UMStn+LOv7GTzEITLzQcblVR1ddPpNBo29/Wyura+sZmYau4vbO7VyrvH7R0nCoOTR7LWHV8pkGKCJooUEInUcBCX0LbH91M6+1HUFrE0T2OE/BCNohEIDhDY/XKJRekrLo4BGRX9OG0V67YNXsmugxODhWSq9Erf7n9mKchRMgl07rr2Al6GVMouIRJ0U01JIyP2AC6BiMWgvay2eETemKcPg1iZV6EdOb+nshYqPU49E1nyHCoF2tT879aN8Xg0stElKQIEZ8vClJJMabTFGhfKOAoxwYYV8LcSvmQKcbRZFU0ITiLX16G1lnNMXx3Xqlf53EUyBE5JlXikAtSJ7ekQZqEk5Q8k1fyZj1ZL9a79TFvXbHymUPyR9bnD4IqklI=</latexit>

Pobs<latexit sha1_base64="hC0T52sOMmnMvUe2iGv5JI7qq7Q=">AAAB8nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdCTFLx4rGBroV1KNs22oclmSWaFsvRnePGgiFd/jTf/jWm7B219IfDwzgyZeaNUCou+/+2V1tY3NrfK25Wd3b39g+rhUdvqzDDeYlpq04mo5VIkvIUCJe+khlMVSf4YjW9n9ccnbqzQyQNOUh4qOkxELBhFZ3Wb/bxnFNGRnfarNb/uz0VWISigBoWa/epXb6BZpniCTFJru4GfYphTg4JJPq30MstTysZ0yLsOE6q4DfP5ylNy5pwBibVxL0Eyd39P5FRZO1GR61QUR3a5NjP/q3UzjK/DXCRphjxhi4/iTBLUZHY/GQjDGcqJA8qMcLsSNqKGMnQpVVwIwfLJq9C+qAeO7y9rjZsijjKcwCmcQwBX0IA7aEILGGh4hld489B78d69j0VryStmjuGPvM8fL52RKg==</latexit><latexit sha1_base64="hC0T52sOMmnMvUe2iGv5JI7qq7Q=">AAAB8nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdCTFLx4rGBroV1KNs22oclmSWaFsvRnePGgiFd/jTf/jWm7B219IfDwzgyZeaNUCou+/+2V1tY3NrfK25Wd3b39g+rhUdvqzDDeYlpq04mo5VIkvIUCJe+khlMVSf4YjW9n9ccnbqzQyQNOUh4qOkxELBhFZ3Wb/bxnFNGRnfarNb/uz0VWISigBoWa/epXb6BZpniCTFJru4GfYphTg4JJPq30MstTysZ0yLsOE6q4DfP5ylNy5pwBibVxL0Eyd39P5FRZO1GR61QUR3a5NjP/q3UzjK/DXCRphjxhi4/iTBLUZHY/GQjDGcqJA8qMcLsSNqKGMnQpVVwIwfLJq9C+qAeO7y9rjZsijjKcwCmcQwBX0IA7aEILGGh4hld489B78d69j0VryStmjuGPvM8fL52RKg==</latexit><latexit sha1_base64="hC0T52sOMmnMvUe2iGv5JI7qq7Q=">AAAB8nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdCTFLx4rGBroV1KNs22oclmSWaFsvRnePGgiFd/jTf/jWm7B219IfDwzgyZeaNUCou+/+2V1tY3NrfK25Wd3b39g+rhUdvqzDDeYlpq04mo5VIkvIUCJe+khlMVSf4YjW9n9ccnbqzQyQNOUh4qOkxELBhFZ3Wb/bxnFNGRnfarNb/uz0VWISigBoWa/epXb6BZpniCTFJru4GfYphTg4JJPq30MstTysZ0yLsOE6q4DfP5ylNy5pwBibVxL0Eyd39P5FRZO1GR61QUR3a5NjP/q3UzjK/DXCRphjxhi4/iTBLUZHY/GQjDGcqJA8qMcLsSNqKGMnQpVVwIwfLJq9C+qAeO7y9rjZsijjKcwCmcQwBX0IA7aEILGGh4hld489B78d69j0VryStmjuGPvM8fL52RKg==</latexit><latexit sha1_base64="hC0T52sOMmnMvUe2iGv5JI7qq7Q=">AAAB8nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdCTFLx4rGBroV1KNs22oclmSWaFsvRnePGgiFd/jTf/jWm7B219IfDwzgyZeaNUCou+/+2V1tY3NrfK25Wd3b39g+rhUdvqzDDeYlpq04mo5VIkvIUCJe+khlMVSf4YjW9n9ccnbqzQyQNOUh4qOkxELBhFZ3Wb/bxnFNGRnfarNb/uz0VWISigBoWa/epXb6BZpniCTFJru4GfYphTg4JJPq30MstTysZ0yLsOE6q4DfP5ylNy5pwBibVxL0Eyd39P5FRZO1GR61QUR3a5NjP/q3UzjK/DXCRphjxhi4/iTBLUZHY/GQjDGcqJA8qMcLsSNqKGMnQpVVwIwfLJq9C+qAeO7y9rjZsijjKcwCmcQwBX0IA7aEILGGh4hld489B78d69j0VryStmjuGPvM8fL52RKg==</latexit>

minimize✓2⇥ EPobs [`(✓;Z)]<latexit sha1_base64="jnG+zUYtV3h0az31g7v7ZxCaHYg=">AAACQnicbZBNSxxBEIZ7TPxav9Z4zKXJIuhFZkRQEGRBhBw34Kpxexh6emvdxu6eobtGWIfxr3nJL8gtP8CLB0PwmkN6dvcQPwoaHt6ql65601xJh2H4K5j58HF2bn5hsbG0vLK61lz/dOaywgroikxl9iLlDpQ00EWJCi5yC1ynCs7T6+O6f34D1snMnOIoh1jzKyMHUnD0UtL8XjKrqZZGankLVVKyocu5gDLKsWI4BOSUSUPZaY1V445pjsM0LU/8bCcZu7PUVVWPgVJbE8chvdyOk2Yr3AnHRd9CNIUWmVYnaf5k/UwUGgwKxZ3rRWGOccktSqGgarDCgV/tml9Bz6PhGlxcjiOo6KZX+nSQWf8M0rH6v6Pk2rmRTv1kfYB73avF93q9AgcHcSlNXiAYMfloUCiKGa3zpH1pQaAaeeDCSr8rFUNuuUCfesOHEL0++S2c7e5Enr/ttdpH0zgWyGfyhWyRiOyTNvlKOqRLBLknD+SJ/A5+BI/Bn+B5MjoTTD0b5EUFf/8BKw6yNg==</latexit><latexit sha1_base64="jnG+zUYtV3h0az31g7v7ZxCaHYg=">AAACQnicbZBNSxxBEIZ7TPxav9Z4zKXJIuhFZkRQEGRBhBw34Kpxexh6emvdxu6eobtGWIfxr3nJL8gtP8CLB0PwmkN6dvcQPwoaHt6ql65601xJh2H4K5j58HF2bn5hsbG0vLK61lz/dOaywgroikxl9iLlDpQ00EWJCi5yC1ynCs7T6+O6f34D1snMnOIoh1jzKyMHUnD0UtL8XjKrqZZGankLVVKyocu5gDLKsWI4BOSUSUPZaY1V445pjsM0LU/8bCcZu7PUVVWPgVJbE8chvdyOk2Yr3AnHRd9CNIUWmVYnaf5k/UwUGgwKxZ3rRWGOccktSqGgarDCgV/tml9Bz6PhGlxcjiOo6KZX+nSQWf8M0rH6v6Pk2rmRTv1kfYB73avF93q9AgcHcSlNXiAYMfloUCiKGa3zpH1pQaAaeeDCSr8rFUNuuUCfesOHEL0++S2c7e5Enr/ttdpH0zgWyGfyhWyRiOyTNvlKOqRLBLknD+SJ/A5+BI/Bn+B5MjoTTD0b5EUFf/8BKw6yNg==</latexit><latexit sha1_base64="jnG+zUYtV3h0az31g7v7ZxCaHYg=">AAACQnicbZBNSxxBEIZ7TPxav9Z4zKXJIuhFZkRQEGRBhBw34Kpxexh6emvdxu6eobtGWIfxr3nJL8gtP8CLB0PwmkN6dvcQPwoaHt6ql65601xJh2H4K5j58HF2bn5hsbG0vLK61lz/dOaywgroikxl9iLlDpQ00EWJCi5yC1ynCs7T6+O6f34D1snMnOIoh1jzKyMHUnD0UtL8XjKrqZZGankLVVKyocu5gDLKsWI4BOSUSUPZaY1V445pjsM0LU/8bCcZu7PUVVWPgVJbE8chvdyOk2Yr3AnHRd9CNIUWmVYnaf5k/UwUGgwKxZ3rRWGOccktSqGgarDCgV/tml9Bz6PhGlxcjiOo6KZX+nSQWf8M0rH6v6Pk2rmRTv1kfYB73avF93q9AgcHcSlNXiAYMfloUCiKGa3zpH1pQaAaeeDCSr8rFUNuuUCfesOHEL0++S2c7e5Enr/ttdpH0zgWyGfyhWyRiOyTNvlKOqRLBLknD+SJ/A5+BI/Bn+B5MjoTTD0b5EUFf/8BKw6yNg==</latexit><latexit sha1_base64="jnG+zUYtV3h0az31g7v7ZxCaHYg=">AAACQnicbZBNSxxBEIZ7TPxav9Z4zKXJIuhFZkRQEGRBhBw34Kpxexh6emvdxu6eobtGWIfxr3nJL8gtP8CLB0PwmkN6dvcQPwoaHt6ql65601xJh2H4K5j58HF2bn5hsbG0vLK61lz/dOaywgroikxl9iLlDpQ00EWJCi5yC1ynCs7T6+O6f34D1snMnOIoh1jzKyMHUnD0UtL8XjKrqZZGankLVVKyocu5gDLKsWI4BOSUSUPZaY1V445pjsM0LU/8bCcZu7PUVVWPgVJbE8chvdyOk2Yr3AnHRd9CNIUWmVYnaf5k/UwUGgwKxZ3rRWGOccktSqGgarDCgV/tml9Bz6PhGlxcjiOo6KZX+nSQWf8M0rH6v6Pk2rmRTv1kfYB73avF93q9AgcHcSlNXiAYMfloUCiKGa3zpH1pQaAaeeDCSr8rFUNuuUCfesOHEL0++S2c7e5Enr/ttdpH0zgWyGfyhWyRiOyTNvlKOqRLBLknD+SJ/A5+BI/Bn+B5MjoTTD0b5EUFf/8BKw6yNg==</latexit>

`(✓;X,Y ) = (1� Y ✓>X)+<latexit sha1_base64="INsm6hx+4nyyKmHU7eNhVdA1IPw=">AAACE3icbZDJSgNBEIZ7XGPcoh69NAYhcQkzIiiIEvDiMYLZyMTQ06kkTXoWumuEEPIOXnwVLx4U8erFm29jZzlo4g8NH39VUV2/F0mh0ba/rbn5hcWl5cRKcnVtfWMztbVd0mGsOBR5KENV8ZgGKQIookAJlUgB8z0JZa97PayXH0BpEQZ32Iug7rN2IFqCMzRWI3XggpQZFzuA7IJWjmg1Sy9pxjmu0rF572IY0Uq2cdhIpe2cPRKdBWcCaTJRoZH6cpshj30IkEumdc2xI6z3mULBJQySbqwhYrzL2lAzGDAfdL0/umlA943TpK1QmRcgHbm/J/rM17rne6bTZ9jR07Wh+V+tFmPrvN4XQRQjBHy8qBVLiiEdBkSbQgFH2TPAuBLmr5R3mGIcTYxJE4IzffIslE5yjuHb03T+ahJHguySPZIhDjkjeXJDCqRIOHkkz+SVvFlP1ov1bn2MW+esycwO+SPr8wcfwJsl</latexit><latexit sha1_base64="INsm6hx+4nyyKmHU7eNhVdA1IPw=">AAACE3icbZDJSgNBEIZ7XGPcoh69NAYhcQkzIiiIEvDiMYLZyMTQ06kkTXoWumuEEPIOXnwVLx4U8erFm29jZzlo4g8NH39VUV2/F0mh0ba/rbn5hcWl5cRKcnVtfWMztbVd0mGsOBR5KENV8ZgGKQIookAJlUgB8z0JZa97PayXH0BpEQZ32Iug7rN2IFqCMzRWI3XggpQZFzuA7IJWjmg1Sy9pxjmu0rF572IY0Uq2cdhIpe2cPRKdBWcCaTJRoZH6cpshj30IkEumdc2xI6z3mULBJQySbqwhYrzL2lAzGDAfdL0/umlA943TpK1QmRcgHbm/J/rM17rne6bTZ9jR07Wh+V+tFmPrvN4XQRQjBHy8qBVLiiEdBkSbQgFH2TPAuBLmr5R3mGIcTYxJE4IzffIslE5yjuHb03T+ahJHguySPZIhDjkjeXJDCqRIOHkkz+SVvFlP1ov1bn2MW+esycwO+SPr8wcfwJsl</latexit><latexit sha1_base64="INsm6hx+4nyyKmHU7eNhVdA1IPw=">AAACE3icbZDJSgNBEIZ7XGPcoh69NAYhcQkzIiiIEvDiMYLZyMTQ06kkTXoWumuEEPIOXnwVLx4U8erFm29jZzlo4g8NH39VUV2/F0mh0ba/rbn5hcWl5cRKcnVtfWMztbVd0mGsOBR5KENV8ZgGKQIookAJlUgB8z0JZa97PayXH0BpEQZ32Iug7rN2IFqCMzRWI3XggpQZFzuA7IJWjmg1Sy9pxjmu0rF572IY0Uq2cdhIpe2cPRKdBWcCaTJRoZH6cpshj30IkEumdc2xI6z3mULBJQySbqwhYrzL2lAzGDAfdL0/umlA943TpK1QmRcgHbm/J/rM17rne6bTZ9jR07Wh+V+tFmPrvN4XQRQjBHy8qBVLiiEdBkSbQgFH2TPAuBLmr5R3mGIcTYxJE4IzffIslE5yjuHb03T+ahJHguySPZIhDjkjeXJDCqRIOHkkz+SVvFlP1ov1bn2MW+esycwO+SPr8wcfwJsl</latexit><latexit sha1_base64="INsm6hx+4nyyKmHU7eNhVdA1IPw=">AAACE3icbZDJSgNBEIZ7XGPcoh69NAYhcQkzIiiIEvDiMYLZyMTQ06kkTXoWumuEEPIOXnwVLx4U8erFm29jZzlo4g8NH39VUV2/F0mh0ba/rbn5hcWl5cRKcnVtfWMztbVd0mGsOBR5KENV8ZgGKQIookAJlUgB8z0JZa97PayXH0BpEQZ32Iug7rN2IFqCMzRWI3XggpQZFzuA7IJWjmg1Sy9pxjmu0rF572IY0Uq2cdhIpe2cPRKdBWcCaTJRoZH6cpshj30IkEumdc2xI6z3mULBJQySbqwhYrzL2lAzGDAfdL0/umlA943TpK1QmRcgHbm/J/rM17rne6bTZ9jR07Wh+V+tFmPrvN4XQRQjBHy8qBVLiiEdBkSbQgFH2TPAuBLmr5R3mGIcTYxJE4IzffIslE5yjuHb03T+ahJHguySPZIhDjkjeXJDCqRIOHkkz+SVvFlP1ov1bn2MW+esycwO+SPr8wcfwJsl</latexit>

✓ 2 ⇥<latexit sha1_base64="cExpTTqhx8J4BpnbJSPS+b3JYSk=">AAAB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosunFZoRehKWUyPWmHTiZh5kQpsY/ixoUibn0Sd76N08tCW38Y+PjPOZwzf5hKYdDzvp3C2vrG5lZxu7Szu7d/4JYPWybJNIcmT2Si70NmQAoFTRQo4T7VwOJQQjsc3Uzr7QfQRiSqgeMUujEbKBEJztBaPbcc4BCQ0UAoGjSm2HMrXtWbia6Cv4AKWajec7+CfsKzGBRyyYzp+F6K3ZxpFFzCpBRkBlLGR2wAHYuKxWC6+ez0CT21Tp9GibZPIZ25vydyFhszjkPbGTMcmuXa1Pyv1skwuurmQqUZguLzRVEmKSZ0mgPtCw0c5dgC41rYWykfMs042rRKNgR/+cur0Dqv+pbvLiq160UcRXJMTsgZ8cklqZFbUidNwskjeSav5M15cl6cd+dj3lpwFjNH5I+czx+VnpOM</latexit><latexit sha1_base64="cExpTTqhx8J4BpnbJSPS+b3JYSk=">AAAB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosunFZoRehKWUyPWmHTiZh5kQpsY/ixoUibn0Sd76N08tCW38Y+PjPOZwzf5hKYdDzvp3C2vrG5lZxu7Szu7d/4JYPWybJNIcmT2Si70NmQAoFTRQo4T7VwOJQQjsc3Uzr7QfQRiSqgeMUujEbKBEJztBaPbcc4BCQ0UAoGjSm2HMrXtWbia6Cv4AKWajec7+CfsKzGBRyyYzp+F6K3ZxpFFzCpBRkBlLGR2wAHYuKxWC6+ez0CT21Tp9GibZPIZ25vydyFhszjkPbGTMcmuXa1Pyv1skwuurmQqUZguLzRVEmKSZ0mgPtCw0c5dgC41rYWykfMs042rRKNgR/+cur0Dqv+pbvLiq160UcRXJMTsgZ8cklqZFbUidNwskjeSav5M15cl6cd+dj3lpwFjNH5I+czx+VnpOM</latexit><latexit sha1_base64="cExpTTqhx8J4BpnbJSPS+b3JYSk=">AAAB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosunFZoRehKWUyPWmHTiZh5kQpsY/ixoUibn0Sd76N08tCW38Y+PjPOZwzf5hKYdDzvp3C2vrG5lZxu7Szu7d/4JYPWybJNIcmT2Si70NmQAoFTRQo4T7VwOJQQjsc3Uzr7QfQRiSqgeMUujEbKBEJztBaPbcc4BCQ0UAoGjSm2HMrXtWbia6Cv4AKWajec7+CfsKzGBRyyYzp+F6K3ZxpFFzCpBRkBlLGR2wAHYuKxWC6+ez0CT21Tp9GibZPIZ25vydyFhszjkPbGTMcmuXa1Pyv1skwuurmQqUZguLzRVEmKSZ0mgPtCw0c5dgC41rYWykfMs042rRKNgR/+cur0Dqv+pbvLiq160UcRXJMTsgZ8cklqZFbUidNwskjeSav5M15cl6cd+dj3lpwFjNH5I+czx+VnpOM</latexit><latexit sha1_base64="cExpTTqhx8J4BpnbJSPS+b3JYSk=">AAAB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosunFZoRehKWUyPWmHTiZh5kQpsY/ixoUibn0Sd76N08tCW38Y+PjPOZwzf5hKYdDzvp3C2vrG5lZxu7Szu7d/4JYPWybJNIcmT2Si70NmQAoFTRQo4T7VwOJQQjsc3Uzr7QfQRiSqgeMUujEbKBEJztBaPbcc4BCQ0UAoGjSm2HMrXtWbia6Cv4AKWajec7+CfsKzGBRyyYzp+F6K3ZxpFFzCpBRkBlLGR2wAHYuKxWC6+ez0CT21Tp9GibZPIZ25vydyFhszjkPbGTMcmuXa1Pyv1skwuurmQqUZguLzRVEmKSZ0mgPtCw0c5dgC41rYWykfMs042rRKNgR/+cur0Dqv+pbvLiq160UcRXJMTsgZ8cklqZFbUidNwskjeSav5M15cl6cd+dj3lpwFjNH5I+czx+VnpOM</latexit>

Z ⇠ Pobs<latexit sha1_base64="p6ncfocElWAw31JKJqx+hZGoGXY=">AAAB+3icbZDLSsNAFIYn9VbrLdalm8EiuCqJCLqSghuXFewFmxAm00k7dC5hZiKWkFdx40IRt76IO9/GaZuFtv4w8PGfczhn/jhlVBvP+3Yqa+sbm1vV7drO7t7+gXtY72qZKUw6WDKp+jHShFFBOoYaRvqpIojHjPTiyc2s3nskSlMp7s00JSFHI0ETipGxVuTWH2CgKYftKA8UhzLWReQ2vKY3F1wFv4QGKNWO3K9gKHHGiTCYIa0HvpeaMEfKUMxIUQsyTVKEJ2hEBhYF4kSH+fz2Ap5aZwgTqewTBs7d3xM54lpPeWw7OTJjvVybmf/VBplJrsKcijQzRODFoiRj0Eg4CwIOqSLYsKkFhBW1t0I8RgphY+Oq2RD85S+vQve86Vu+u2i0rss4quAYnIAz4INL0AK3oA06AIMn8AxewZtTOC/Ou/OxaK045cwR+CPn8wcx4ZPg</latexit><latexit sha1_base64="p6ncfocElWAw31JKJqx+hZGoGXY=">AAAB+3icbZDLSsNAFIYn9VbrLdalm8EiuCqJCLqSghuXFewFmxAm00k7dC5hZiKWkFdx40IRt76IO9/GaZuFtv4w8PGfczhn/jhlVBvP+3Yqa+sbm1vV7drO7t7+gXtY72qZKUw6WDKp+jHShFFBOoYaRvqpIojHjPTiyc2s3nskSlMp7s00JSFHI0ETipGxVuTWH2CgKYftKA8UhzLWReQ2vKY3F1wFv4QGKNWO3K9gKHHGiTCYIa0HvpeaMEfKUMxIUQsyTVKEJ2hEBhYF4kSH+fz2Ap5aZwgTqewTBs7d3xM54lpPeWw7OTJjvVybmf/VBplJrsKcijQzRODFoiRj0Eg4CwIOqSLYsKkFhBW1t0I8RgphY+Oq2RD85S+vQve86Vu+u2i0rss4quAYnIAz4INL0AK3oA06AIMn8AxewZtTOC/Ou/OxaK045cwR+CPn8wcx4ZPg</latexit><latexit sha1_base64="p6ncfocElWAw31JKJqx+hZGoGXY=">AAAB+3icbZDLSsNAFIYn9VbrLdalm8EiuCqJCLqSghuXFewFmxAm00k7dC5hZiKWkFdx40IRt76IO9/GaZuFtv4w8PGfczhn/jhlVBvP+3Yqa+sbm1vV7drO7t7+gXtY72qZKUw6WDKp+jHShFFBOoYaRvqpIojHjPTiyc2s3nskSlMp7s00JSFHI0ETipGxVuTWH2CgKYftKA8UhzLWReQ2vKY3F1wFv4QGKNWO3K9gKHHGiTCYIa0HvpeaMEfKUMxIUQsyTVKEJ2hEBhYF4kSH+fz2Ap5aZwgTqewTBs7d3xM54lpPeWw7OTJjvVybmf/VBplJrsKcijQzRODFoiRj0Eg4CwIOqSLYsKkFhBW1t0I8RgphY+Oq2RD85S+vQve86Vu+u2i0rss4quAYnIAz4INL0AK3oA06AIMn8AxewZtTOC/Ou/OxaK045cwR+CPn8wcx4ZPg</latexit><latexit sha1_base64="p6ncfocElWAw31JKJqx+hZGoGXY=">AAAB+3icbZDLSsNAFIYn9VbrLdalm8EiuCqJCLqSghuXFewFmxAm00k7dC5hZiKWkFdx40IRt76IO9/GaZuFtv4w8PGfczhn/jhlVBvP+3Yqa+sbm1vV7drO7t7+gXtY72qZKUw6WDKp+jHShFFBOoYaRvqpIojHjPTiyc2s3nskSlMp7s00JSFHI0ETipGxVuTWH2CgKYftKA8UhzLWReQ2vKY3F1wFv4QGKNWO3K9gKHHGiTCYIa0HvpeaMEfKUMxIUQsyTVKEJ2hEBhYF4kSH+fz2Ap5aZwgTqewTBs7d3xM54lpPeWw7OTJjvVybmf/VBplJrsKcijQzRODFoiRj0Eg4CwIOqSLYsKkFhBW1t0I8RgphY+Oq2RD85S+vQve86Vu+u2i0rss4quAYnIAz4INL0AK3oA06AIMn8AxewZtTOC/Ou/OxaK045cwR+CPn8wcx4ZPg</latexit>

`(✓;X,Y ) = (Y � ✓>X)2<latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit>

`(✓;X,Y ) = (Y � �1(✓1 · · ·�k(✓k ·X))))2<latexit sha1_base64="PftD3QrTS8T4CA+d9G8Hp2Bh7Pk=">AAACO3icbZBLSwMxFIUzvq2vqks3wSK0oGWmCAqiFNy4rGK1pVOHTHrbhmYeJHeEUvxfbvwT7ty4caGIW/em7QhaPRA4nO+G5B4/lkKjbT9ZU9Mzs3PzC4uZpeWV1bXs+saVjhLFocojGamazzRIEUIVBUqoxQpY4Eu49nunQ359C0qLKLzEfgzNgHVC0RacoYm87IULUuZd7AKyI1rbpfUCPab5Ot2jrhadgHlOSj2HurwVof4GvW/QGwNaKxjdlLxszi7aI9G/xklNjqSqeNlHtxXxJIAQuWRaNxw7xuaAKRRcwl3GTTTEjPdYBxrGhiwA3RyMdr+jOyZp0XakzAmRjtKfNwYs0Lof+GYyYNjVk2wY/scaCbYPmwMRxglCyMcPtRNJMaLDImlLKOAo+8YwroT5K+VdphhHU3fGlOBMrvzXXJWKjvHn+7nySVrHAtki2yRPHHJAyuSMVEiVcHJPnskrebMerBfr3foYj05Z6Z1N8kvW5xen66pH</latexit><latexit sha1_base64="PftD3QrTS8T4CA+d9G8Hp2Bh7Pk=">AAACO3icbZBLSwMxFIUzvq2vqks3wSK0oGWmCAqiFNy4rGK1pVOHTHrbhmYeJHeEUvxfbvwT7ty4caGIW/em7QhaPRA4nO+G5B4/lkKjbT9ZU9Mzs3PzC4uZpeWV1bXs+saVjhLFocojGamazzRIEUIVBUqoxQpY4Eu49nunQ359C0qLKLzEfgzNgHVC0RacoYm87IULUuZd7AKyI1rbpfUCPab5Ot2jrhadgHlOSj2HurwVof4GvW/QGwNaKxjdlLxszi7aI9G/xklNjqSqeNlHtxXxJIAQuWRaNxw7xuaAKRRcwl3GTTTEjPdYBxrGhiwA3RyMdr+jOyZp0XakzAmRjtKfNwYs0Lof+GYyYNjVk2wY/scaCbYPmwMRxglCyMcPtRNJMaLDImlLKOAo+8YwroT5K+VdphhHU3fGlOBMrvzXXJWKjvHn+7nySVrHAtki2yRPHHJAyuSMVEiVcHJPnskrebMerBfr3foYj05Z6Z1N8kvW5xen66pH</latexit><latexit sha1_base64="PftD3QrTS8T4CA+d9G8Hp2Bh7Pk=">AAACO3icbZBLSwMxFIUzvq2vqks3wSK0oGWmCAqiFNy4rGK1pVOHTHrbhmYeJHeEUvxfbvwT7ty4caGIW/em7QhaPRA4nO+G5B4/lkKjbT9ZU9Mzs3PzC4uZpeWV1bXs+saVjhLFocojGamazzRIEUIVBUqoxQpY4Eu49nunQ359C0qLKLzEfgzNgHVC0RacoYm87IULUuZd7AKyI1rbpfUCPab5Ot2jrhadgHlOSj2HurwVof4GvW/QGwNaKxjdlLxszi7aI9G/xklNjqSqeNlHtxXxJIAQuWRaNxw7xuaAKRRcwl3GTTTEjPdYBxrGhiwA3RyMdr+jOyZp0XakzAmRjtKfNwYs0Lof+GYyYNjVk2wY/scaCbYPmwMRxglCyMcPtRNJMaLDImlLKOAo+8YwroT5K+VdphhHU3fGlOBMrvzXXJWKjvHn+7nySVrHAtki2yRPHHJAyuSMVEiVcHJPnskrebMerBfr3foYj05Z6Z1N8kvW5xen66pH</latexit><latexit sha1_base64="PftD3QrTS8T4CA+d9G8Hp2Bh7Pk=">AAACO3icbZBLSwMxFIUzvq2vqks3wSK0oGWmCAqiFNy4rGK1pVOHTHrbhmYeJHeEUvxfbvwT7ty4caGIW/em7QhaPRA4nO+G5B4/lkKjbT9ZU9Mzs3PzC4uZpeWV1bXs+saVjhLFocojGamazzRIEUIVBUqoxQpY4Eu49nunQ359C0qLKLzEfgzNgHVC0RacoYm87IULUuZd7AKyI1rbpfUCPab5Ot2jrhadgHlOSj2HurwVof4GvW/QGwNaKxjdlLxszi7aI9G/xklNjqSqeNlHtxXxJIAQuWRaNxw7xuaAKRRcwl3GTTTEjPdYBxrGhiwA3RyMdr+jOyZp0XakzAmRjtKfNwYs0Lof+GYyYNjVk2wY/scaCbYPmwMRxglCyMcPtRNJMaLDImlLKOAo+8YwroT5K+VdphhHU3fGlOBMrvzXXJWKjvHn+7nySVrHAtki2yRPHHJAyuSMVEiVcHJPnskrebMerBfr3foYj05Z6Z1N8kvW5xen66pH</latexit>

Page 15: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Example: Facial Recognition• Labeled Faces in the Wild, a gold standard dataset for face

recognition, is 77.5% male, and 83.5% White [Han and Jain ’14]

• Commercial gender classification softwares have disparate performance on different subpopulations

Gendered Shades: Intersectional accuracy disparity[Buolamwini and Gebru ’18]

Page 16: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Fair predictions?7JHOFUUF �� 'BJS QSFEJDUJPO NPEFMT

0.0

0.5

1.0

1.5

2.0

Aver

age

Erro

r

Model A Model B

Atypical examples are important for fairness

Average Loss for a Minority

Loss

Den

sity

WorseBetter

Examples:

Video Captioning[Tatman+ 17]

Recommender systems[Ekstrand+ 17,18]

Dependency parsing[Blodgett+ 16]

Page 17: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

The challenges of interpolation

Saturday Morning Breakfast Cereal,Zach Wienersmith

Page 18: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• In some applications, demographic information is unavailable (e.g. speech recognition), or illegal to use (e.g. insurance)

• Protected groups are hard to define a priori- variables often comprise continuous spectrum (e.g. skin color)- performance determined in an intersectional fashion

• Accounting for intersections gives exponentially many subgroups- computational & statistical difficulties

First Idea: Pre-defined groups

Problems

[Meinshausen & Buhlmann ’15]

• Separate model for each group

• One model for worst-off group

EPg [`(✓g;Z)]<latexit sha1_base64="jzOmPhG0DCJvWWtuPNbYR32mg/8=">AAACOHicbZDLSsNAFIYnXus96tLNYBHqQklEUHBTEcFlBatiEsJketoOTi7MnAglBHwL30Vwq8/gzp24E5/A6WWh1gMDP//5h3POF2VSaHScV2ticmp6ZrYyN7+wuLS8Yq+uXeo0VxyaPJWpuo6YBikSaKJACdeZAhZHEq6i25N+/+oOlBZpcoG9DIKYdRLRFpyhsULb9WOG3SgqTsuw8BW0ikbYKUvq+SBlzY9kDoWPXUBm7CN6sx2EdtXZdQZFx4U7ElUyqkZof/qtlOcxJMgl09pznQyDgikUXEI57+caMsZvWQc8IxMWgw6KwWkl3TJOi7ZTZV6CdOD+/FGwWOteHJlk/xD9t9c3/+t5ObYPg0IkWY6Q8OGgdi4pprTPibaEAo6yZwTjSphdKe8yxTgamr+maDQstQnondhEU10aRO5fIOPicm/XNfp8v1o/vh/CqpANsklqxCUHpE7OSIM0CScP5Ik8kxfr0Xqz3q2PYXTCGgFeJ7/K+voGtSeu4A==</latexit><latexit sha1_base64="jzOmPhG0DCJvWWtuPNbYR32mg/8=">AAACOHicbZDLSsNAFIYnXus96tLNYBHqQklEUHBTEcFlBatiEsJketoOTi7MnAglBHwL30Vwq8/gzp24E5/A6WWh1gMDP//5h3POF2VSaHScV2ticmp6ZrYyN7+wuLS8Yq+uXeo0VxyaPJWpuo6YBikSaKJACdeZAhZHEq6i25N+/+oOlBZpcoG9DIKYdRLRFpyhsULb9WOG3SgqTsuw8BW0ikbYKUvq+SBlzY9kDoWPXUBm7CN6sx2EdtXZdQZFx4U7ElUyqkZof/qtlOcxJMgl09pznQyDgikUXEI57+caMsZvWQc8IxMWgw6KwWkl3TJOi7ZTZV6CdOD+/FGwWOteHJlk/xD9t9c3/+t5ObYPg0IkWY6Q8OGgdi4pprTPibaEAo6yZwTjSphdKe8yxTgamr+maDQstQnondhEU10aRO5fIOPicm/XNfp8v1o/vh/CqpANsklqxCUHpE7OSIM0CScP5Ik8kxfr0Xqz3q2PYXTCGgFeJ7/K+voGtSeu4A==</latexit><latexit sha1_base64="jzOmPhG0DCJvWWtuPNbYR32mg/8=">AAACOHicbZDLSsNAFIYnXus96tLNYBHqQklEUHBTEcFlBatiEsJketoOTi7MnAglBHwL30Vwq8/gzp24E5/A6WWh1gMDP//5h3POF2VSaHScV2ticmp6ZrYyN7+wuLS8Yq+uXeo0VxyaPJWpuo6YBikSaKJACdeZAhZHEq6i25N+/+oOlBZpcoG9DIKYdRLRFpyhsULb9WOG3SgqTsuw8BW0ikbYKUvq+SBlzY9kDoWPXUBm7CN6sx2EdtXZdQZFx4U7ElUyqkZof/qtlOcxJMgl09pznQyDgikUXEI57+caMsZvWQc8IxMWgw6KwWkl3TJOi7ZTZV6CdOD+/FGwWOteHJlk/xD9t9c3/+t5ObYPg0IkWY6Q8OGgdi4pprTPibaEAo6yZwTjSphdKe8yxTgamr+maDQstQnondhEU10aRO5fIOPicm/XNfp8v1o/vh/CqpANsklqxCUHpE7OSIM0CScP5Ik8kxfr0Xqz3q2PYXTCGgFeJ7/K+voGtSeu4A==</latexit><latexit sha1_base64="jzOmPhG0DCJvWWtuPNbYR32mg/8=">AAACOHicbZDLSsNAFIYnXus96tLNYBHqQklEUHBTEcFlBatiEsJketoOTi7MnAglBHwL30Vwq8/gzp24E5/A6WWh1gMDP//5h3POF2VSaHScV2ticmp6ZrYyN7+wuLS8Yq+uXeo0VxyaPJWpuo6YBikSaKJACdeZAhZHEq6i25N+/+oOlBZpcoG9DIKYdRLRFpyhsULb9WOG3SgqTsuw8BW0ikbYKUvq+SBlzY9kDoWPXUBm7CN6sx2EdtXZdQZFx4U7ElUyqkZof/qtlOcxJMgl09pznQyDgikUXEI57+caMsZvWQc8IxMWgw6KwWkl3TJOi7ZTZV6CdOD+/FGwWOteHJlk/xD9t9c3/+t5ObYPg0IkWY6Q8OGgdi4pprTPibaEAo6yZwTjSphdKe8yxTgamr+maDQstQnondhEU10aRO5fIOPicm/XNfp8v1o/vh/CqpANsklqxCUHpE7OSIM0CScP5Ik8kxfr0Xqz3q2PYXTCGgFeJ7/K+voGtSeu4A==</latexit>

maxg2G

EPg [`(✓;Z)]<latexit sha1_base64="s5An/auV4//sY0hZcONkr7S77gk=">AAACVnicbZDfStxAFMYnsXbV/jHVS2+GLgV7UUlKQcEbRUTv3EJXpZsQJrNnd4edTMLMSXEZIr6U72Lprd73CUpns7mo2gMD33znG86cX1ZKYTAMf3r+0ovll52V1bVXr9+8XQ/ebZybotIc+ryQhb7MmAEpFPRRoITLUgPLMwkX2fRo3r/4AdqIQn3DWQlJzsZKjARn6Kw0OItzdpXaWMPQjmksFHUGTjiT9qSua3rdXLPMHtdtqpeOnT+IQcrtOJMV2BgngKzep98/JmnQDXfCpuhzEbWiS9rqpcHveFjwKgeFXDJjBlFYYmKZRsEl1GtxZaBkfMrGMHBSsRxMYpvFa/rBOUM6KrQ7Cmnj/vvCstyYWZ655HwN87Q3N//XG1Q42kusUGWFoPhi0KiSFAs6p0iHQgNHOXOCcS3cXymfMM04OtaPphh0pI0LmE+5ixamdoiip0Cei/PPO5HTX790Dw5vFrBWyBZ5T7ZJRHbJATklPdInnNySX+SePHh33h9/2e8sor7XAt4kj8oP/gLTTri+</latexit><latexit sha1_base64="s5An/auV4//sY0hZcONkr7S77gk=">AAACVnicbZDfStxAFMYnsXbV/jHVS2+GLgV7UUlKQcEbRUTv3EJXpZsQJrNnd4edTMLMSXEZIr6U72Lprd73CUpns7mo2gMD33znG86cX1ZKYTAMf3r+0ovll52V1bVXr9+8XQ/ebZybotIc+ryQhb7MmAEpFPRRoITLUgPLMwkX2fRo3r/4AdqIQn3DWQlJzsZKjARn6Kw0OItzdpXaWMPQjmksFHUGTjiT9qSua3rdXLPMHtdtqpeOnT+IQcrtOJMV2BgngKzep98/JmnQDXfCpuhzEbWiS9rqpcHveFjwKgeFXDJjBlFYYmKZRsEl1GtxZaBkfMrGMHBSsRxMYpvFa/rBOUM6KrQ7Cmnj/vvCstyYWZ655HwN87Q3N//XG1Q42kusUGWFoPhi0KiSFAs6p0iHQgNHOXOCcS3cXymfMM04OtaPphh0pI0LmE+5ixamdoiip0Cei/PPO5HTX790Dw5vFrBWyBZ5T7ZJRHbJATklPdInnNySX+SePHh33h9/2e8sor7XAt4kj8oP/gLTTri+</latexit><latexit sha1_base64="s5An/auV4//sY0hZcONkr7S77gk=">AAACVnicbZDfStxAFMYnsXbV/jHVS2+GLgV7UUlKQcEbRUTv3EJXpZsQJrNnd4edTMLMSXEZIr6U72Lprd73CUpns7mo2gMD33znG86cX1ZKYTAMf3r+0ovll52V1bVXr9+8XQ/ebZybotIc+ryQhb7MmAEpFPRRoITLUgPLMwkX2fRo3r/4AdqIQn3DWQlJzsZKjARn6Kw0OItzdpXaWMPQjmksFHUGTjiT9qSua3rdXLPMHtdtqpeOnT+IQcrtOJMV2BgngKzep98/JmnQDXfCpuhzEbWiS9rqpcHveFjwKgeFXDJjBlFYYmKZRsEl1GtxZaBkfMrGMHBSsRxMYpvFa/rBOUM6KrQ7Cmnj/vvCstyYWZ655HwN87Q3N//XG1Q42kusUGWFoPhi0KiSFAs6p0iHQgNHOXOCcS3cXymfMM04OtaPphh0pI0LmE+5ixamdoiip0Cei/PPO5HTX790Dw5vFrBWyBZ5T7ZJRHbJATklPdInnNySX+SePHh33h9/2e8sor7XAt4kj8oP/gLTTri+</latexit><latexit sha1_base64="s5An/auV4//sY0hZcONkr7S77gk=">AAACVnicbZDfStxAFMYnsXbV/jHVS2+GLgV7UUlKQcEbRUTv3EJXpZsQJrNnd4edTMLMSXEZIr6U72Lprd73CUpns7mo2gMD33znG86cX1ZKYTAMf3r+0ovll52V1bVXr9+8XQ/ebZybotIc+ryQhb7MmAEpFPRRoITLUgPLMwkX2fRo3r/4AdqIQn3DWQlJzsZKjARn6Kw0OItzdpXaWMPQjmksFHUGTjiT9qSua3rdXLPMHtdtqpeOnT+IQcrtOJMV2BgngKzep98/JmnQDXfCpuhzEbWiS9rqpcHveFjwKgeFXDJjBlFYYmKZRsEl1GtxZaBkfMrGMHBSsRxMYpvFa/rBOUM6KrQ7Cmnj/vvCstyYWZ655HwN87Q3N//XG1Q42kusUGWFoPhi0KiSFAs6p0iHQgNHOXOCcS3cXymfMM04OtaPphh0pI0LmE+5ixamdoiip0Cei/PPO5HTX790Dw5vFrBWyBZ5T7ZJRHbJATklPdInnNySX+SePHh33h9/2e8sor7XAt4kj8oP/gLTTri+</latexit>

Given pre-defined demographic groups g 2 G,<latexit sha1_base64="dLwHv1i08B3biXZWJYOWIcz7psk=">AAACH3icbZDLSsNAFIYn9VbrLepGcDNYBBdaEhF0WXGhywr2Ak0ok8mkHTozCTMToYSIryK41ddwJ277Fj6Ck7YLWz0w8POffzjnfEHCqNKOM7ZKS8srq2vl9crG5tb2jr2711JxKjFp4pjFshMgRRgVpKmpZqSTSIJ4wEg7GN4U/fYjkYrG4kGPEuJz1Bc0ohhpY/XsA0+SMOtDjwrocaQHGLHsNs9Pe3bVqTmTgn+FOxNVMKtGz/72whinnAiNGVKq6zqJ9jMkNcWM5BUvVSRBeIj6pGukQJwoP5tckMNj44QwiqV5QsOJ+/tHhrhSIx6YZLGkWuwV5n+9bqqjKz+jIkk1EXg6KEoZ1DEscMCQSoI1GxmBsKRmV4gHSCKsDbS5KUobZMoE1Bk30VjlBpG7COSvaJ3XXKPvL6r16+cprDI4BEfgBLjgEtTBHWiAJsDgCbyCN/BuvVgf1qf1NY2WrBngfTBX1vgH5xCkQg==</latexit><latexit sha1_base64="dLwHv1i08B3biXZWJYOWIcz7psk=">AAACH3icbZDLSsNAFIYn9VbrLepGcDNYBBdaEhF0WXGhywr2Ak0ok8mkHTozCTMToYSIryK41ddwJ277Fj6Ck7YLWz0w8POffzjnfEHCqNKOM7ZKS8srq2vl9crG5tb2jr2711JxKjFp4pjFshMgRRgVpKmpZqSTSIJ4wEg7GN4U/fYjkYrG4kGPEuJz1Bc0ohhpY/XsA0+SMOtDjwrocaQHGLHsNs9Pe3bVqTmTgn+FOxNVMKtGz/72whinnAiNGVKq6zqJ9jMkNcWM5BUvVSRBeIj6pGukQJwoP5tckMNj44QwiqV5QsOJ+/tHhrhSIx6YZLGkWuwV5n+9bqqjKz+jIkk1EXg6KEoZ1DEscMCQSoI1GxmBsKRmV4gHSCKsDbS5KUobZMoE1Bk30VjlBpG7COSvaJ3XXKPvL6r16+cprDI4BEfgBLjgEtTBHWiAJsDgCbyCN/BuvVgf1qf1NY2WrBngfTBX1vgH5xCkQg==</latexit><latexit sha1_base64="dLwHv1i08B3biXZWJYOWIcz7psk=">AAACH3icbZDLSsNAFIYn9VbrLepGcDNYBBdaEhF0WXGhywr2Ak0ok8mkHTozCTMToYSIryK41ddwJ277Fj6Ck7YLWz0w8POffzjnfEHCqNKOM7ZKS8srq2vl9crG5tb2jr2711JxKjFp4pjFshMgRRgVpKmpZqSTSIJ4wEg7GN4U/fYjkYrG4kGPEuJz1Bc0ohhpY/XsA0+SMOtDjwrocaQHGLHsNs9Pe3bVqTmTgn+FOxNVMKtGz/72whinnAiNGVKq6zqJ9jMkNcWM5BUvVSRBeIj6pGukQJwoP5tckMNj44QwiqV5QsOJ+/tHhrhSIx6YZLGkWuwV5n+9bqqjKz+jIkk1EXg6KEoZ1DEscMCQSoI1GxmBsKRmV4gHSCKsDbS5KUobZMoE1Bk30VjlBpG7COSvaJ3XXKPvL6r16+cprDI4BEfgBLjgEtTBHWiAJsDgCbyCN/BuvVgf1qf1NY2WrBngfTBX1vgH5xCkQg==</latexit><latexit sha1_base64="dLwHv1i08B3biXZWJYOWIcz7psk=">AAACH3icbZDLSsNAFIYn9VbrLepGcDNYBBdaEhF0WXGhywr2Ak0ok8mkHTozCTMToYSIryK41ddwJ277Fj6Ck7YLWz0w8POffzjnfEHCqNKOM7ZKS8srq2vl9crG5tb2jr2711JxKjFp4pjFshMgRRgVpKmpZqSTSIJ4wEg7GN4U/fYjkYrG4kGPEuJz1Bc0ohhpY/XsA0+SMOtDjwrocaQHGLHsNs9Pe3bVqTmTgn+FOxNVMKtGz/72whinnAiNGVKq6zqJ9jMkNcWM5BUvVSRBeIj6pGukQJwoP5tckMNj44QwiqV5QsOJ+/tHhrhSIx6YZLGkWuwV5n+9bqqjKz+jIkk1EXg6KEoZ1DEscMCQSoI1GxmBsKRmV4gHSCKsDbS5KUobZMoE1Bk30VjlBpG7COSvaJ3XXKPvL6r16+cprDI4BEfgBLjgEtTBHWiAJsDgCbyCN/BuvVgf1qf1NY2WrBngfTBX1vgH5xCkQg==</latexit>

See also [Kearns et al. ’18, Kim et al. ’19]

cf. Constraint-based fairness notions [Calders & Verwer ’10, Zemel ’15, Feldman et al. ’15, Hardt et al. ’16, llJohnson et al. ’16, Chouldechova ’17] and many others

Page 19: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• Guarantee uniform performance across subpopulations

• Computationally efficient

• Characterize statistical price of subpopulation performance

PreviewAutomatically find worst-off subpopulations,

and optimize performance on them

Page 20: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Risk Aversion

pdf(L)<latexit sha1_base64="lmzBfxRbdSd2HwJVSnIN3imQ/70=">AAACEnicbZDLSgMxGIUz9VbrrerSTbAIdWGZEUGXFTcuXFSwF2iHkslk2tBcxiQjlmHAhxDc6mu4E7e+gG/hI5heFrb1h8DhnBP+ny+IGdXGdb+d3NLyyupafr2wsbm1vVPc3WtomShM6lgyqVoB0oRRQeqGGkZasSKIB4w0g8HVKG8+EKWpFHdmGBOfo56gEcXIWMvv8EA+pnEYZeWb426x5Fbc8cBF4U1FCUyn1i3+dEKJE06EwQxp3fbc2PgpUoZiRrJCJ9EkRniAeqRtpUCcaD8dH53BI+uEMJLKPmHg2P37I0Vc6yEPbJMj09fz2cj8L2snJrrwUyrixBCBJ4uihEEj4YgADKki2LChFQgram+FuI8UwsZymtmijaWkbUGfcFuVOrOIvHkgi6JxWvGsvj0rVS+fJrDy4AAcgjLwwDmogmtQA3WAwT14Aa/gzXl23p0P53NSzTlTwPtgZpyvX78Un3o=</latexit><latexit sha1_base64="lmzBfxRbdSd2HwJVSnIN3imQ/70=">AAACEnicbZDLSgMxGIUz9VbrrerSTbAIdWGZEUGXFTcuXFSwF2iHkslk2tBcxiQjlmHAhxDc6mu4E7e+gG/hI5heFrb1h8DhnBP+ny+IGdXGdb+d3NLyyupafr2wsbm1vVPc3WtomShM6lgyqVoB0oRRQeqGGkZasSKIB4w0g8HVKG8+EKWpFHdmGBOfo56gEcXIWMvv8EA+pnEYZeWb426x5Fbc8cBF4U1FCUyn1i3+dEKJE06EwQxp3fbc2PgpUoZiRrJCJ9EkRniAeqRtpUCcaD8dH53BI+uEMJLKPmHg2P37I0Vc6yEPbJMj09fz2cj8L2snJrrwUyrixBCBJ4uihEEj4YgADKki2LChFQgram+FuI8UwsZymtmijaWkbUGfcFuVOrOIvHkgi6JxWvGsvj0rVS+fJrDy4AAcgjLwwDmogmtQA3WAwT14Aa/gzXl23p0P53NSzTlTwPtgZpyvX78Un3o=</latexit><latexit sha1_base64="lmzBfxRbdSd2HwJVSnIN3imQ/70=">AAACEnicbZDLSgMxGIUz9VbrrerSTbAIdWGZEUGXFTcuXFSwF2iHkslk2tBcxiQjlmHAhxDc6mu4E7e+gG/hI5heFrb1h8DhnBP+ny+IGdXGdb+d3NLyyupafr2wsbm1vVPc3WtomShM6lgyqVoB0oRRQeqGGkZasSKIB4w0g8HVKG8+EKWpFHdmGBOfo56gEcXIWMvv8EA+pnEYZeWb426x5Fbc8cBF4U1FCUyn1i3+dEKJE06EwQxp3fbc2PgpUoZiRrJCJ9EkRniAeqRtpUCcaD8dH53BI+uEMJLKPmHg2P37I0Vc6yEPbJMj09fz2cj8L2snJrrwUyrixBCBJ4uihEEj4YgADKki2LChFQgram+FuI8UwsZymtmijaWkbUGfcFuVOrOIvHkgi6JxWvGsvj0rVS+fJrDy4AAcgjLwwDmogmtQA3WAwT14Aa/gzXl23p0P53NSzTlTwPtgZpyvX78Un3o=</latexit><latexit sha1_base64="lmzBfxRbdSd2HwJVSnIN3imQ/70=">AAACEnicbZDLSgMxGIUz9VbrrerSTbAIdWGZEUGXFTcuXFSwF2iHkslk2tBcxiQjlmHAhxDc6mu4E7e+gG/hI5heFrb1h8DhnBP+ny+IGdXGdb+d3NLyyupafr2wsbm1vVPc3WtomShM6lgyqVoB0oRRQeqGGkZasSKIB4w0g8HVKG8+EKWpFHdmGBOfo56gEcXIWMvv8EA+pnEYZeWb426x5Fbc8cBF4U1FCUyn1i3+dEKJE06EwQxp3fbc2PgpUoZiRrJCJ9EkRniAeqRtpUCcaD8dH53BI+uEMJLKPmHg2P37I0Vc6yEPbJMj09fz2cj8L2snJrrwUyrixBCBJ4uihEEj4YgADKki2LChFQgram+FuI8UwsZymtmijaWkbUGfcFuVOrOIvHkgi6JxWvGsvj0rVS+fJrDy4AAcgjLwwDmogmtQA3WAwT14Aa/gzXl23p0P53NSzTlTwPtgZpyvX78Un3o=</latexit>

E[`(✓1;Z)] � E[`(✓2;Z)]<latexit sha1_base64="HKr4iwn6tfX+UTAo2kYrhlCRr5s=">AAACPnicbVDLSgNBEJz1/Tbq0ctgEPSg7AZBwYsogkcFo2J2CbOTTjI4u7NM9wphyaf4L4JX/QB/wJvo0aOTB/gsGCiqqumeijOtkHz/2RsZHRufmJyanpmdm19YLC0tX6DJrYSqNNrYq1ggaJVClRRpuMosiCTWcBnfHPX8y1uwqEx6Tp0MokS0UtVUUpCT6qXd8LgWaoO4EVpoFCG1gUQ96O7z682Ihy3gX4GBV+lb9VLZ3/b74H9JMCRlNsRpvfQeNozME0hJaoFYC/yMokJYUlJDdybMETIhb0QLao6mIgGMiv4Hu3zdKQ3eNNa9lHhf/T5RiASxk8QumQhq42+vJ/7n1XJq7kWFSrOcIJWDRc1cczK81xZvKAuSdMcRIa1yt3LZFlZIcp3+2ILkGkUXwK3ERQ12XUXB70L+kovKduD42U754HBY1hRbZWtsgwVslx2wE3bKqkyyO/bAHtmTd++9eK/e2yA64g1nVtgPeB+fJhqu4w==</latexit><latexit sha1_base64="HKr4iwn6tfX+UTAo2kYrhlCRr5s=">AAACPnicbVDLSgNBEJz1/Tbq0ctgEPSg7AZBwYsogkcFo2J2CbOTTjI4u7NM9wphyaf4L4JX/QB/wJvo0aOTB/gsGCiqqumeijOtkHz/2RsZHRufmJyanpmdm19YLC0tX6DJrYSqNNrYq1ggaJVClRRpuMosiCTWcBnfHPX8y1uwqEx6Tp0MokS0UtVUUpCT6qXd8LgWaoO4EVpoFCG1gUQ96O7z682Ihy3gX4GBV+lb9VLZ3/b74H9JMCRlNsRpvfQeNozME0hJaoFYC/yMokJYUlJDdybMETIhb0QLao6mIgGMiv4Hu3zdKQ3eNNa9lHhf/T5RiASxk8QumQhq42+vJ/7n1XJq7kWFSrOcIJWDRc1cczK81xZvKAuSdMcRIa1yt3LZFlZIcp3+2ILkGkUXwK3ERQ12XUXB70L+kovKduD42U754HBY1hRbZWtsgwVslx2wE3bKqkyyO/bAHtmTd++9eK/e2yA64g1nVtgPeB+fJhqu4w==</latexit><latexit sha1_base64="HKr4iwn6tfX+UTAo2kYrhlCRr5s=">AAACPnicbVDLSgNBEJz1/Tbq0ctgEPSg7AZBwYsogkcFo2J2CbOTTjI4u7NM9wphyaf4L4JX/QB/wJvo0aOTB/gsGCiqqumeijOtkHz/2RsZHRufmJyanpmdm19YLC0tX6DJrYSqNNrYq1ggaJVClRRpuMosiCTWcBnfHPX8y1uwqEx6Tp0MokS0UtVUUpCT6qXd8LgWaoO4EVpoFCG1gUQ96O7z682Ihy3gX4GBV+lb9VLZ3/b74H9JMCRlNsRpvfQeNozME0hJaoFYC/yMokJYUlJDdybMETIhb0QLao6mIgGMiv4Hu3zdKQ3eNNa9lHhf/T5RiASxk8QumQhq42+vJ/7n1XJq7kWFSrOcIJWDRc1cczK81xZvKAuSdMcRIa1yt3LZFlZIcp3+2ILkGkUXwK3ERQ12XUXB70L+kovKduD42U754HBY1hRbZWtsgwVslx2wE3bKqkyyO/bAHtmTd++9eK/e2yA64g1nVtgPeB+fJhqu4w==</latexit><latexit sha1_base64="HKr4iwn6tfX+UTAo2kYrhlCRr5s=">AAACPnicbVDLSgNBEJz1/Tbq0ctgEPSg7AZBwYsogkcFo2J2CbOTTjI4u7NM9wphyaf4L4JX/QB/wJvo0aOTB/gsGCiqqumeijOtkHz/2RsZHRufmJyanpmdm19YLC0tX6DJrYSqNNrYq1ggaJVClRRpuMosiCTWcBnfHPX8y1uwqEx6Tp0MokS0UtVUUpCT6qXd8LgWaoO4EVpoFCG1gUQ96O7z682Ihy3gX4GBV+lb9VLZ3/b74H9JMCRlNsRpvfQeNozME0hJaoFYC/yMokJYUlJDdybMETIhb0QLao6mIgGMiv4Hu3zdKQ3eNNa9lHhf/T5RiASxk8QumQhq42+vJ/7n1XJq7kWFSrOcIJWDRc1cczK81xZvKAuSdMcRIa1yt3LZFlZIcp3+2ILkGkUXwK3ERQ12XUXB70L+kovKduD42U754HBY1hRbZWtsgwVslx2wE3bKqkyyO/bAHtmTd++9eK/e2yA64g1nVtgPeB+fJhqu4w==</latexit>

Tail[`(✓1;Z)] Tail[`(✓2;Z)]<latexit sha1_base64="pouPGL+Vaxq3vSQKwQ5C32UjkQU=">AAACWHicbVBNSxxBEK2dfLhqTDbmmEuTRdCDMiNCArlIcvGo4Kq4Myw9vbVuY/f00FUTsgyzf8vfouDVXPMT0vtxiB8PGh6vXvGqX14aTRzHt63o1es3b1faq2vr7zbef+h83DwjV3mFPeWM8xe5JDS6wB5rNnhRepQ2N3ieX/+czc9/oSftilOelJhZeVXokVaSgzTonKQ2d7/rU6lN00+NI9pOPQ7rlMfIcpA038XlTiamC9FgMxUvbCzM+3PvoNON9+I5xHOSLEkXljgedP6mQ6cqiwUrI4n6SVxyVkvPWoXAtbQiLKW6llfYD7SQFimr519vxFZQhmLkfHgFi7n6/0YtLdHE5sFpJY/p6WwmvjTrVzz6ltW6KCvGQi2CRpUR7MSsRzHUHhWbSSBSeR1uFWosvVQc2n6UQhy6pmCgXRusjppQUfK0kOfkbH8vCfzkoHv4Y1lWGz7DF9iGBL7CIRzBMfRAwQ3cwwP8ad1FEK1Eqwtr1FrufIJHiDb/ARe0t2E=</latexit><latexit sha1_base64="pouPGL+Vaxq3vSQKwQ5C32UjkQU=">AAACWHicbVBNSxxBEK2dfLhqTDbmmEuTRdCDMiNCArlIcvGo4Kq4Myw9vbVuY/f00FUTsgyzf8vfouDVXPMT0vtxiB8PGh6vXvGqX14aTRzHt63o1es3b1faq2vr7zbef+h83DwjV3mFPeWM8xe5JDS6wB5rNnhRepQ2N3ieX/+czc9/oSftilOelJhZeVXokVaSgzTonKQ2d7/rU6lN00+NI9pOPQ7rlMfIcpA038XlTiamC9FgMxUvbCzM+3PvoNON9+I5xHOSLEkXljgedP6mQ6cqiwUrI4n6SVxyVkvPWoXAtbQiLKW6llfYD7SQFimr519vxFZQhmLkfHgFi7n6/0YtLdHE5sFpJY/p6WwmvjTrVzz6ltW6KCvGQi2CRpUR7MSsRzHUHhWbSSBSeR1uFWosvVQc2n6UQhy6pmCgXRusjppQUfK0kOfkbH8vCfzkoHv4Y1lWGz7DF9iGBL7CIRzBMfRAwQ3cwwP8ad1FEK1Eqwtr1FrufIJHiDb/ARe0t2E=</latexit><latexit sha1_base64="pouPGL+Vaxq3vSQKwQ5C32UjkQU=">AAACWHicbVBNSxxBEK2dfLhqTDbmmEuTRdCDMiNCArlIcvGo4Kq4Myw9vbVuY/f00FUTsgyzf8vfouDVXPMT0vtxiB8PGh6vXvGqX14aTRzHt63o1es3b1faq2vr7zbef+h83DwjV3mFPeWM8xe5JDS6wB5rNnhRepQ2N3ieX/+czc9/oSftilOelJhZeVXokVaSgzTonKQ2d7/rU6lN00+NI9pOPQ7rlMfIcpA038XlTiamC9FgMxUvbCzM+3PvoNON9+I5xHOSLEkXljgedP6mQ6cqiwUrI4n6SVxyVkvPWoXAtbQiLKW6llfYD7SQFimr519vxFZQhmLkfHgFi7n6/0YtLdHE5sFpJY/p6WwmvjTrVzz6ltW6KCvGQi2CRpUR7MSsRzHUHhWbSSBSeR1uFWosvVQc2n6UQhy6pmCgXRusjppQUfK0kOfkbH8vCfzkoHv4Y1lWGz7DF9iGBL7CIRzBMfRAwQ3cwwP8ad1FEK1Eqwtr1FrufIJHiDb/ARe0t2E=</latexit><latexit sha1_base64="pouPGL+Vaxq3vSQKwQ5C32UjkQU=">AAACWHicbVBNSxxBEK2dfLhqTDbmmEuTRdCDMiNCArlIcvGo4Kq4Myw9vbVuY/f00FUTsgyzf8vfouDVXPMT0vtxiB8PGh6vXvGqX14aTRzHt63o1es3b1faq2vr7zbef+h83DwjV3mFPeWM8xe5JDS6wB5rNnhRepQ2N3ieX/+czc9/oSftilOelJhZeVXokVaSgzTonKQ2d7/rU6lN00+NI9pOPQ7rlMfIcpA038XlTiamC9FgMxUvbCzM+3PvoNON9+I5xHOSLEkXljgedP6mQ6cqiwUrI4n6SVxyVkvPWoXAtbQiLKW6llfYD7SQFimr519vxFZQhmLkfHgFi7n6/0YtLdHE5sFpJY/p6WwmvjTrVzz6ltW6KCvGQi2CRpUR7MSsRzHUHhWbSSBSeR1uFWosvVQc2n6UQhy6pmCgXRusjppQUfK0kOfkbH8vCfzkoHv4Y1lWGz7DF9iGBL7CIRzBMfRAwQ3cwwP8ad1FEK1Eqwtr1FrufIJHiDb/ARe0t2E=</latexit>

`(✓1;Z)<latexit sha1_base64="XFKVJY86x7ZNg/oUmuGU11IASUM=">AAACH3icbZDNSgMxFIUz/lv/qm4EN8Ei6EKZEUHBTcWNSwVri51SMumtDWYmQ+4doQwVX0Vwq6/hTtz6Fj6C6c/CqhcCh3NOuMkXpVoh+f6nNzE5NT0zOzdfWFhcWl4prq5do8mshIo02thaJBC0SqBCijTUUgsijjRUo7uzfl69B4vKJFfUTaERi9tEtZUU5KxmcSPUBnEntNDKQ+oAiWbQO+E3u81iyd/3B8P/imAkSmw0F83iV9gyMoshIakFYj3wU2rkwpKSGnqFMENIhbwTt1B3MhExYCMf/KDHt53T4m1j3UmID9yfN3IRI3bjyDVjQR38nfXN/7J6Ru3jRq6SNCNI5HBRO9OcDO/j4C1lQZLuOiGkVe6tXHaEFZIctLEtSA4ZugLuxa5qsOcQBb+B/BXXB/uB05eHpfLp4xDWHNtkW2yHBeyIldk5u2AVJtkDe2Yv7NV78t68d+9jWJ3wRoDX2dh4n99efaPs</latexit><latexit sha1_base64="XFKVJY86x7ZNg/oUmuGU11IASUM=">AAACH3icbZDNSgMxFIUz/lv/qm4EN8Ei6EKZEUHBTcWNSwVri51SMumtDWYmQ+4doQwVX0Vwq6/hTtz6Fj6C6c/CqhcCh3NOuMkXpVoh+f6nNzE5NT0zOzdfWFhcWl4prq5do8mshIo02thaJBC0SqBCijTUUgsijjRUo7uzfl69B4vKJFfUTaERi9tEtZUU5KxmcSPUBnEntNDKQ+oAiWbQO+E3u81iyd/3B8P/imAkSmw0F83iV9gyMoshIakFYj3wU2rkwpKSGnqFMENIhbwTt1B3MhExYCMf/KDHt53T4m1j3UmID9yfN3IRI3bjyDVjQR38nfXN/7J6Ru3jRq6SNCNI5HBRO9OcDO/j4C1lQZLuOiGkVe6tXHaEFZIctLEtSA4ZugLuxa5qsOcQBb+B/BXXB/uB05eHpfLp4xDWHNtkW2yHBeyIldk5u2AVJtkDe2Yv7NV78t68d+9jWJ3wRoDX2dh4n99efaPs</latexit><latexit sha1_base64="XFKVJY86x7ZNg/oUmuGU11IASUM=">AAACH3icbZDNSgMxFIUz/lv/qm4EN8Ei6EKZEUHBTcWNSwVri51SMumtDWYmQ+4doQwVX0Vwq6/hTtz6Fj6C6c/CqhcCh3NOuMkXpVoh+f6nNzE5NT0zOzdfWFhcWl4prq5do8mshIo02thaJBC0SqBCijTUUgsijjRUo7uzfl69B4vKJFfUTaERi9tEtZUU5KxmcSPUBnEntNDKQ+oAiWbQO+E3u81iyd/3B8P/imAkSmw0F83iV9gyMoshIakFYj3wU2rkwpKSGnqFMENIhbwTt1B3MhExYCMf/KDHt53T4m1j3UmID9yfN3IRI3bjyDVjQR38nfXN/7J6Ru3jRq6SNCNI5HBRO9OcDO/j4C1lQZLuOiGkVe6tXHaEFZIctLEtSA4ZugLuxa5qsOcQBb+B/BXXB/uB05eHpfLp4xDWHNtkW2yHBeyIldk5u2AVJtkDe2Yv7NV78t68d+9jWJ3wRoDX2dh4n99efaPs</latexit><latexit sha1_base64="XFKVJY86x7ZNg/oUmuGU11IASUM=">AAACH3icbZDNSgMxFIUz/lv/qm4EN8Ei6EKZEUHBTcWNSwVri51SMumtDWYmQ+4doQwVX0Vwq6/hTtz6Fj6C6c/CqhcCh3NOuMkXpVoh+f6nNzE5NT0zOzdfWFhcWl4prq5do8mshIo02thaJBC0SqBCijTUUgsijjRUo7uzfl69B4vKJFfUTaERi9tEtZUU5KxmcSPUBnEntNDKQ+oAiWbQO+E3u81iyd/3B8P/imAkSmw0F83iV9gyMoshIakFYj3wU2rkwpKSGnqFMENIhbwTt1B3MhExYCMf/KDHt53T4m1j3UmID9yfN3IRI3bjyDVjQR38nfXN/7J6Ru3jRq6SNCNI5HBRO9OcDO/j4C1lQZLuOiGkVe6tXHaEFZIctLEtSA4ZugLuxa5qsOcQBb+B/BXXB/uB05eHpfLp4xDWHNtkW2yHBeyIldk5u2AVJtkDe2Yv7NV78t68d+9jWJ3wRoDX2dh4n99efaPs</latexit>`(✓2;Z)

<latexit sha1_base64="skEQkUNAPHNZIVeRnejIm3INrs8=">AAACGXicbZDLSgMxFIYz9Vbrrdalm2AR6sIyUwQFNxU3LivYC7alZNK0Dc1MhpwzYikF30Nwq6/hTty68i18BNPLwrYeCPz8/x/O4fMjJQFd99tJrKyurW8kN1Nb2zu7e+n9TAV0bLgoc620qfkMhJKhKKNEJWqRESzwlaj6/etxXn0QBqQO73AQiWbAuqHsSM7QWq10pqE0QK6BPYGsVbik9yetdNbNu5Ohy8KbiSyZTamV/mm0NY8DESJXDKDuuRE2h8yg5EqMUo0YRMR4n3VF3cqQBQKaw8ntI3psnTbtaGNfiHTi/v0xZAHAIPBtM2DYg8VsbP6X1WPsXDSHMoxiFCGfLurEiqKmYxC0LY3gqAZWMG6kvZXyHjOMo8U1twXQwgJbgNPAVjWMLCJvEciyqBTyntW3Z9ni1dMUVpIckiOSIx45J0VyQ0qkTDh5JC/klbw5z8678+F8TqsJZwb4gMyN8/ULITihIg==</latexit><latexit sha1_base64="skEQkUNAPHNZIVeRnejIm3INrs8=">AAACGXicbZDLSgMxFIYz9Vbrrdalm2AR6sIyUwQFNxU3LivYC7alZNK0Dc1MhpwzYikF30Nwq6/hTty68i18BNPLwrYeCPz8/x/O4fMjJQFd99tJrKyurW8kN1Nb2zu7e+n9TAV0bLgoc620qfkMhJKhKKNEJWqRESzwlaj6/etxXn0QBqQO73AQiWbAuqHsSM7QWq10pqE0QK6BPYGsVbik9yetdNbNu5Ohy8KbiSyZTamV/mm0NY8DESJXDKDuuRE2h8yg5EqMUo0YRMR4n3VF3cqQBQKaw8ntI3psnTbtaGNfiHTi/v0xZAHAIPBtM2DYg8VsbP6X1WPsXDSHMoxiFCGfLurEiqKmYxC0LY3gqAZWMG6kvZXyHjOMo8U1twXQwgJbgNPAVjWMLCJvEciyqBTyntW3Z9ni1dMUVpIckiOSIx45J0VyQ0qkTDh5JC/klbw5z8678+F8TqsJZwb4gMyN8/ULITihIg==</latexit><latexit sha1_base64="skEQkUNAPHNZIVeRnejIm3INrs8=">AAACGXicbZDLSgMxFIYz9Vbrrdalm2AR6sIyUwQFNxU3LivYC7alZNK0Dc1MhpwzYikF30Nwq6/hTty68i18BNPLwrYeCPz8/x/O4fMjJQFd99tJrKyurW8kN1Nb2zu7e+n9TAV0bLgoc620qfkMhJKhKKNEJWqRESzwlaj6/etxXn0QBqQO73AQiWbAuqHsSM7QWq10pqE0QK6BPYGsVbik9yetdNbNu5Ohy8KbiSyZTamV/mm0NY8DESJXDKDuuRE2h8yg5EqMUo0YRMR4n3VF3cqQBQKaw8ntI3psnTbtaGNfiHTi/v0xZAHAIPBtM2DYg8VsbP6X1WPsXDSHMoxiFCGfLurEiqKmYxC0LY3gqAZWMG6kvZXyHjOMo8U1twXQwgJbgNPAVjWMLCJvEciyqBTyntW3Z9ni1dMUVpIckiOSIx45J0VyQ0qkTDh5JC/klbw5z8678+F8TqsJZwb4gMyN8/ULITihIg==</latexit><latexit sha1_base64="skEQkUNAPHNZIVeRnejIm3INrs8=">AAACGXicbZDLSgMxFIYz9Vbrrdalm2AR6sIyUwQFNxU3LivYC7alZNK0Dc1MhpwzYikF30Nwq6/hTty68i18BNPLwrYeCPz8/x/O4fMjJQFd99tJrKyurW8kN1Nb2zu7e+n9TAV0bLgoc620qfkMhJKhKKNEJWqRESzwlaj6/etxXn0QBqQO73AQiWbAuqHsSM7QWq10pqE0QK6BPYGsVbik9yetdNbNu5Ohy8KbiSyZTamV/mm0NY8DESJXDKDuuRE2h8yg5EqMUo0YRMR4n3VF3cqQBQKaw8ntI3psnTbtaGNfiHTi/v0xZAHAIPBtM2DYg8VsbP6X1WPsXDSHMoxiFCGfLurEiqKmYxC0LY3gqAZWMG6kvZXyHjOMo8U1twXQwgJbgNPAVjWMLCJvEciyqBTyntW3Z9ni1dMUVpIckiOSIx45J0VyQ0qkTDh5JC/klbw5z8678+F8TqsJZwb4gMyN8/ULITihIg==</latexit>

Risk-aversion: prefer over . ✓1<latexit sha1_base64="CJASpDG60VeOY6LsZRKxQyBLiSY=">AAACE3icbZDNSsNAFIUn9a/Wv6pLN8EiuLEkIuiy4sZlBfsDbSyTyU07dCYJMzdCCQVfQnCrr+FO3PoAvoWP4KTtwrZeGDicc4Z7+fxEcI2O820VVlbX1jeKm6Wt7Z3dvfL+QVPHqWLQYLGIVdunGgSPoIEcBbQTBVT6Alr+8CbPW4+gNI+jexwl4Enaj3jIGUVjPXQVBFkXB4C054575YpTdSZjLwt3JipkNvVe+acbxCyVECETVOuO6yToZVQhZwLGpW6qIaFsSPvQMTKiErSXTa4e2yfGCewwVuZFaE/cvz8yKrUeSd80JcWBXsxy87+sk2J45WU8SlKEiE0XhamwMbZzBHbAFTAUIyMoU9zcarMBVZShATW3RaPBpE1Bn0lTjXWOyF0Esiya51XX6LuLSu36aQqrSI7IMTklLrkkNXJL6qRBGFHkhbySN+vZerc+rM9ptWDNAB+SubG+fgEIg6A0</latexit><latexit sha1_base64="CJASpDG60VeOY6LsZRKxQyBLiSY=">AAACE3icbZDNSsNAFIUn9a/Wv6pLN8EiuLEkIuiy4sZlBfsDbSyTyU07dCYJMzdCCQVfQnCrr+FO3PoAvoWP4KTtwrZeGDicc4Z7+fxEcI2O820VVlbX1jeKm6Wt7Z3dvfL+QVPHqWLQYLGIVdunGgSPoIEcBbQTBVT6Alr+8CbPW4+gNI+jexwl4Enaj3jIGUVjPXQVBFkXB4C054575YpTdSZjLwt3JipkNvVe+acbxCyVECETVOuO6yToZVQhZwLGpW6qIaFsSPvQMTKiErSXTa4e2yfGCewwVuZFaE/cvz8yKrUeSd80JcWBXsxy87+sk2J45WU8SlKEiE0XhamwMbZzBHbAFTAUIyMoU9zcarMBVZShATW3RaPBpE1Bn0lTjXWOyF0Esiya51XX6LuLSu36aQqrSI7IMTklLrkkNXJL6qRBGFHkhbySN+vZerc+rM9ptWDNAB+SubG+fgEIg6A0</latexit><latexit sha1_base64="CJASpDG60VeOY6LsZRKxQyBLiSY=">AAACE3icbZDNSsNAFIUn9a/Wv6pLN8EiuLEkIuiy4sZlBfsDbSyTyU07dCYJMzdCCQVfQnCrr+FO3PoAvoWP4KTtwrZeGDicc4Z7+fxEcI2O820VVlbX1jeKm6Wt7Z3dvfL+QVPHqWLQYLGIVdunGgSPoIEcBbQTBVT6Alr+8CbPW4+gNI+jexwl4Enaj3jIGUVjPXQVBFkXB4C054575YpTdSZjLwt3JipkNvVe+acbxCyVECETVOuO6yToZVQhZwLGpW6qIaFsSPvQMTKiErSXTa4e2yfGCewwVuZFaE/cvz8yKrUeSd80JcWBXsxy87+sk2J45WU8SlKEiE0XhamwMbZzBHbAFTAUIyMoU9zcarMBVZShATW3RaPBpE1Bn0lTjXWOyF0Esiya51XX6LuLSu36aQqrSI7IMTklLrkkNXJL6qRBGFHkhbySN+vZerc+rM9ptWDNAB+SubG+fgEIg6A0</latexit><latexit sha1_base64="CJASpDG60VeOY6LsZRKxQyBLiSY=">AAACE3icbZDNSsNAFIUn9a/Wv6pLN8EiuLEkIuiy4sZlBfsDbSyTyU07dCYJMzdCCQVfQnCrr+FO3PoAvoWP4KTtwrZeGDicc4Z7+fxEcI2O820VVlbX1jeKm6Wt7Z3dvfL+QVPHqWLQYLGIVdunGgSPoIEcBbQTBVT6Alr+8CbPW4+gNI+jexwl4Enaj3jIGUVjPXQVBFkXB4C054575YpTdSZjLwt3JipkNvVe+acbxCyVECETVOuO6yToZVQhZwLGpW6qIaFsSPvQMTKiErSXTa4e2yfGCewwVuZFaE/cvz8yKrUeSd80JcWBXsxy87+sk2J45WU8SlKEiE0XhamwMbZzBHbAFTAUIyMoU9zcarMBVZShATW3RaPBpE1Bn0lTjXWOyF0Esiya51XX6LuLSu36aQqrSI7IMTklLrkkNXJL6qRBGFHkhbySN+vZerc+rM9ptWDNAB+SubG+fgEIg6A0</latexit>

✓2<latexit sha1_base64="K2VQKYZu5yOAPxvrhXSkPi8Q0cI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scwUQZcVNy4r2Ae0Q8mkmTY0jyG5I5Sh4B8IbvU33Ilbv8G/8BNMHwvbeiBwOOeEe++JEsEt+P63l1tb39jcym8Xdnb39g+Kh0cNq1NDWZ1qoU0rIpYJrlgdOAjWSgwjMhKsGQ1vJ37zkRnLtXqAUcJCSfqKx5wScFKrAwMGpFvpFkt+2Z8Cr5JgTkpojlq3+NPpaZpKpoAKYm078BMIM2KAU8HGhU5qWULokPRZ21FFJLNhNt13jM+c0sOxNu4pwFP174+MSGtHMnJJSWBgl72J+J/XTiG+DjOukhSYorNBcSowaDw5Hve4YRTEyBFCDXe7YjoghlBwFS1MseAKsi5gL6SLajt2FQXLhaySRqUcOH5/WarePM3KyqMTdIrOUYCuUBXdoRqqI4oEekGv6M179t69D+9zFs1584KP0QK8r1/kN51q</latexit><latexit sha1_base64="K2VQKYZu5yOAPxvrhXSkPi8Q0cI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scwUQZcVNy4r2Ae0Q8mkmTY0jyG5I5Sh4B8IbvU33Ilbv8G/8BNMHwvbeiBwOOeEe++JEsEt+P63l1tb39jcym8Xdnb39g+Kh0cNq1NDWZ1qoU0rIpYJrlgdOAjWSgwjMhKsGQ1vJ37zkRnLtXqAUcJCSfqKx5wScFKrAwMGpFvpFkt+2Z8Cr5JgTkpojlq3+NPpaZpKpoAKYm078BMIM2KAU8HGhU5qWULokPRZ21FFJLNhNt13jM+c0sOxNu4pwFP174+MSGtHMnJJSWBgl72J+J/XTiG+DjOukhSYorNBcSowaDw5Hve4YRTEyBFCDXe7YjoghlBwFS1MseAKsi5gL6SLajt2FQXLhaySRqUcOH5/WarePM3KyqMTdIrOUYCuUBXdoRqqI4oEekGv6M179t69D+9zFs1584KP0QK8r1/kN51q</latexit><latexit sha1_base64="K2VQKYZu5yOAPxvrhXSkPi8Q0cI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scwUQZcVNy4r2Ae0Q8mkmTY0jyG5I5Sh4B8IbvU33Ilbv8G/8BNMHwvbeiBwOOeEe++JEsEt+P63l1tb39jcym8Xdnb39g+Kh0cNq1NDWZ1qoU0rIpYJrlgdOAjWSgwjMhKsGQ1vJ37zkRnLtXqAUcJCSfqKx5wScFKrAwMGpFvpFkt+2Z8Cr5JgTkpojlq3+NPpaZpKpoAKYm078BMIM2KAU8HGhU5qWULokPRZ21FFJLNhNt13jM+c0sOxNu4pwFP174+MSGtHMnJJSWBgl72J+J/XTiG+DjOukhSYorNBcSowaDw5Hve4YRTEyBFCDXe7YjoghlBwFS1MseAKsi5gL6SLajt2FQXLhaySRqUcOH5/WarePM3KyqMTdIrOUYCuUBXdoRqqI4oEekGv6M179t69D+9zFs1584KP0QK8r1/kN51q</latexit><latexit sha1_base64="K2VQKYZu5yOAPxvrhXSkPi8Q0cI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scwUQZcVNy4r2Ae0Q8mkmTY0jyG5I5Sh4B8IbvU33Ilbv8G/8BNMHwvbeiBwOOeEe++JEsEt+P63l1tb39jcym8Xdnb39g+Kh0cNq1NDWZ1qoU0rIpYJrlgdOAjWSgwjMhKsGQ1vJ37zkRnLtXqAUcJCSfqKx5wScFKrAwMGpFvpFkt+2Z8Cr5JgTkpojlq3+NPpaZpKpoAKYm078BMIM2KAU8HGhU5qWULokPRZ21FFJLNhNt13jM+c0sOxNu4pwFP174+MSGtHMnJJSWBgl72J+J/XTiG+DjOukhSYorNBcSowaDw5Hve4YRTEyBFCDXe7YjoghlBwFS1MseAKsi5gL6SLajt2FQXLhaySRqUcOH5/WarePM3KyqMTdIrOUYCuUBXdoRqqI4oEekGv6M179t69D+9zFs1584KP0QK8r1/kN51q</latexit>

Page 21: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Conditional Value-at-Risk

[Rockafellar and Uryasev ’00]

(1� ↵)-quantile of `(✓;Z)<latexit sha1_base64="hpRXB15bA3rAtL1CbfsgI/p7+jk=">AAACPHicbZBLaxRBFIWr4yNxfI26dFM4CJPFDN2iGHATceMygpMEp4fhds3tTJF6tHVvi0PT4v/wvwhu9Re4dyeCK9fWPBYm8ULB4ZxT3KqvqIwmTtPvydaly1eubu9c61y/cfPW7e6du4fk66BwpLzx4bgAQqMdjlizweMqINjC4FFx+mKZH73DQNq717yocGLhxOlSK+BoTbtP8oCzpp8NcjDVHHbb3Bb+fTN4W4NjbVD6sv2QG0/Uz3mODM/km93OtNtLh+lq5EWRbURPbOZg2v2dz7yqLTpWBojGWVrxpIHAWhlsO3lNWIE6hRMcR+nAIk2a1fda+TA6M1n6EI9juXL/vdGAJVrYIjYt8JzOZ0vzf9m45nJv0mhX1YxOrReVtZHs5ZKVnOmAis0iClBBx7dKNYcAiiPRM1uII0+KBRrYWPXURkTZeSAXxeGjYZYNs1ePe/vPP65h7Yj74oHoi0w8FfvipTgQI6HEJ/FFfBXfks/Jj+Rn8mtd3Uo2gO+JM5P8+QuAyq+n</latexit><latexit sha1_base64="hpRXB15bA3rAtL1CbfsgI/p7+jk=">AAACPHicbZBLaxRBFIWr4yNxfI26dFM4CJPFDN2iGHATceMygpMEp4fhds3tTJF6tHVvi0PT4v/wvwhu9Re4dyeCK9fWPBYm8ULB4ZxT3KqvqIwmTtPvydaly1eubu9c61y/cfPW7e6du4fk66BwpLzx4bgAQqMdjlizweMqINjC4FFx+mKZH73DQNq717yocGLhxOlSK+BoTbtP8oCzpp8NcjDVHHbb3Bb+fTN4W4NjbVD6sv2QG0/Uz3mODM/km93OtNtLh+lq5EWRbURPbOZg2v2dz7yqLTpWBojGWVrxpIHAWhlsO3lNWIE6hRMcR+nAIk2a1fda+TA6M1n6EI9juXL/vdGAJVrYIjYt8JzOZ0vzf9m45nJv0mhX1YxOrReVtZHs5ZKVnOmAis0iClBBx7dKNYcAiiPRM1uII0+KBRrYWPXURkTZeSAXxeGjYZYNs1ePe/vPP65h7Yj74oHoi0w8FfvipTgQI6HEJ/FFfBXfks/Jj+Rn8mtd3Uo2gO+JM5P8+QuAyq+n</latexit><latexit sha1_base64="hpRXB15bA3rAtL1CbfsgI/p7+jk=">AAACPHicbZBLaxRBFIWr4yNxfI26dFM4CJPFDN2iGHATceMygpMEp4fhds3tTJF6tHVvi0PT4v/wvwhu9Re4dyeCK9fWPBYm8ULB4ZxT3KqvqIwmTtPvydaly1eubu9c61y/cfPW7e6du4fk66BwpLzx4bgAQqMdjlizweMqINjC4FFx+mKZH73DQNq717yocGLhxOlSK+BoTbtP8oCzpp8NcjDVHHbb3Bb+fTN4W4NjbVD6sv2QG0/Uz3mODM/km93OtNtLh+lq5EWRbURPbOZg2v2dz7yqLTpWBojGWVrxpIHAWhlsO3lNWIE6hRMcR+nAIk2a1fda+TA6M1n6EI9juXL/vdGAJVrYIjYt8JzOZ0vzf9m45nJv0mhX1YxOrReVtZHs5ZKVnOmAis0iClBBx7dKNYcAiiPRM1uII0+KBRrYWPXURkTZeSAXxeGjYZYNs1ePe/vPP65h7Yj74oHoi0w8FfvipTgQI6HEJ/FFfBXfks/Jj+Rn8mtd3Uo2gO+JM5P8+QuAyq+n</latexit><latexit sha1_base64="hpRXB15bA3rAtL1CbfsgI/p7+jk=">AAACPHicbZBLaxRBFIWr4yNxfI26dFM4CJPFDN2iGHATceMygpMEp4fhds3tTJF6tHVvi0PT4v/wvwhu9Re4dyeCK9fWPBYm8ULB4ZxT3KqvqIwmTtPvydaly1eubu9c61y/cfPW7e6du4fk66BwpLzx4bgAQqMdjlizweMqINjC4FFx+mKZH73DQNq717yocGLhxOlSK+BoTbtP8oCzpp8NcjDVHHbb3Bb+fTN4W4NjbVD6sv2QG0/Uz3mODM/km93OtNtLh+lq5EWRbURPbOZg2v2dz7yqLTpWBojGWVrxpIHAWhlsO3lNWIE6hRMcR+nAIk2a1fda+TA6M1n6EI9juXL/vdGAJVrYIjYt8JzOZ0vzf9m45nJv0mhX1YxOrReVtZHs5ZKVnOmAis0iClBBx7dKNYcAiiPRM1uII0+KBRrYWPXURkTZeSAXxeGjYZYNs1ePe/vPP65h7Yj74oHoi0w8FfvipTgQI6HEJ/FFfBXfks/Jj+Rn8mtd3Uo2gO+JM5P8+QuAyq+n</latexit>

`(✓;Z)<latexit sha1_base64="VzBnNA+MS+yEpOVfDpk1cQpQCzE=">AAACF3icbVDLSgNBEJz1GeNr1aOXwSDEg2FXBAUvQS8eI5gHJiHMTmaTIbM7y3RvIIR8iOBVf8ObePXoX/gJTh4Hk1jQUFRV000FiZKAnvftrKyurW9sZray2zu7e/vuwWEFdGq4KHOttKkFDISSsSijRCVqiREsCpSoBr27sV/tCwNSx484SEQzYp1YhpIztFLLdRtKA+Qb2BXIbujTWcvNeQVvArpM/BnJkRlKLfen0dY8jUSMXDGAuu8l2Bwyg5IrMco2UhAJ4z3WEXVLYxYJaA4nn4/oqVXaNNTGTox0ov7dGLIIYBAFNhkx7MKiNxb/8+ophtfNoYyTFEXMp4fCVFHUdFwDbUsjOKqBJYwbaX+lvMsM42jLmrsCaKsCG4DzyEY1jGxF/mIhy6RyUfAtf7jMFW9nZWXIMTkheeKTK1Ik96REyoSTPnkhr+TNeXbenQ/ncxpdcWY7R2QOztcvuBSf/g==</latexit><latexit sha1_base64="VzBnNA+MS+yEpOVfDpk1cQpQCzE=">AAACF3icbVDLSgNBEJz1GeNr1aOXwSDEg2FXBAUvQS8eI5gHJiHMTmaTIbM7y3RvIIR8iOBVf8ObePXoX/gJTh4Hk1jQUFRV000FiZKAnvftrKyurW9sZray2zu7e/vuwWEFdGq4KHOttKkFDISSsSijRCVqiREsCpSoBr27sV/tCwNSx484SEQzYp1YhpIztFLLdRtKA+Qb2BXIbujTWcvNeQVvArpM/BnJkRlKLfen0dY8jUSMXDGAuu8l2Bwyg5IrMco2UhAJ4z3WEXVLYxYJaA4nn4/oqVXaNNTGTox0ov7dGLIIYBAFNhkx7MKiNxb/8+ophtfNoYyTFEXMp4fCVFHUdFwDbUsjOKqBJYwbaX+lvMsM42jLmrsCaKsCG4DzyEY1jGxF/mIhy6RyUfAtf7jMFW9nZWXIMTkheeKTK1Ik96REyoSTPnkhr+TNeXbenQ/ncxpdcWY7R2QOztcvuBSf/g==</latexit><latexit sha1_base64="VzBnNA+MS+yEpOVfDpk1cQpQCzE=">AAACF3icbVDLSgNBEJz1GeNr1aOXwSDEg2FXBAUvQS8eI5gHJiHMTmaTIbM7y3RvIIR8iOBVf8ObePXoX/gJTh4Hk1jQUFRV000FiZKAnvftrKyurW9sZray2zu7e/vuwWEFdGq4KHOttKkFDISSsSijRCVqiREsCpSoBr27sV/tCwNSx484SEQzYp1YhpIztFLLdRtKA+Qb2BXIbujTWcvNeQVvArpM/BnJkRlKLfen0dY8jUSMXDGAuu8l2Bwyg5IrMco2UhAJ4z3WEXVLYxYJaA4nn4/oqVXaNNTGTox0ov7dGLIIYBAFNhkx7MKiNxb/8+ophtfNoYyTFEXMp4fCVFHUdFwDbUsjOKqBJYwbaX+lvMsM42jLmrsCaKsCG4DzyEY1jGxF/mIhy6RyUfAtf7jMFW9nZWXIMTkheeKTK1Ik96REyoSTPnkhr+TNeXbenQ/ncxpdcWY7R2QOztcvuBSf/g==</latexit><latexit sha1_base64="VzBnNA+MS+yEpOVfDpk1cQpQCzE=">AAACF3icbVDLSgNBEJz1GeNr1aOXwSDEg2FXBAUvQS8eI5gHJiHMTmaTIbM7y3RvIIR8iOBVf8ObePXoX/gJTh4Hk1jQUFRV000FiZKAnvftrKyurW9sZray2zu7e/vuwWEFdGq4KHOttKkFDISSsSijRCVqiREsCpSoBr27sV/tCwNSx484SEQzYp1YhpIztFLLdRtKA+Qb2BXIbujTWcvNeQVvArpM/BnJkRlKLfen0dY8jUSMXDGAuu8l2Bwyg5IrMco2UhAJ4z3WEXVLYxYJaA4nn4/oqVXaNNTGTox0ov7dGLIIYBAFNhkx7MKiNxb/8+ophtfNoYyTFEXMp4fCVFHUdFwDbUsjOKqBJYwbaX+lvMsM42jLmrsCaKsCG4DzyEY1jGxF/mIhy6RyUfAtf7jMFW9nZWXIMTkheeKTK1Ik96REyoSTPnkhr+TNeXbenQ/ncxpdcWY7R2QOztcvuBSf/g==</latexit>

• CVaR defines a tail-average after the -quantile(1� ↵)<latexit sha1_base64="2QtFLFbiPIjQhyO4MxS4aXLrHLw=">AAACD3icbVDLSgMxFM34rPVVdekmWIS6aJkRQZcVNy4r2Ae2Q7mTpm1oHkOSEcpQ8BcEt/ob7sStn+Bf+Ammj4VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdnKbu/s7u3nDg5rRiWa0CpRXOlGBIZyJmnVMstpI9YURMRpPRrcjP36I9WGKXlvhzENBfQk6zIC1kkPhaDYAh734aydy/slfwK8TIIZyaMZKu3cT6ujSCKotISDMc3Aj22YgraMcDrKthJDYyAD6NGmoxIENWE62XiET53SwV2l3ZMWT9S/P1IQxgxF5JICbN8semPxP6+Z2O5VmDIZJ5ZKMh3UTTi2Co/Pxx2mKbF86AgQzdyumPRBA7GupLkpxrqKjAuYonBRZUauomCxkGVSOy8FQSm4u8iXr5+mZWXQMTpBBRSgS1RGt6iCqoggiV7QK3rznr1378P7nEZXvFnBR2gO3tcvRqqdjQ==</latexit><latexit sha1_base64="2QtFLFbiPIjQhyO4MxS4aXLrHLw=">AAACD3icbVDLSgMxFM34rPVVdekmWIS6aJkRQZcVNy4r2Ae2Q7mTpm1oHkOSEcpQ8BcEt/ob7sStn+Bf+Ammj4VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdnKbu/s7u3nDg5rRiWa0CpRXOlGBIZyJmnVMstpI9YURMRpPRrcjP36I9WGKXlvhzENBfQk6zIC1kkPhaDYAh734aydy/slfwK8TIIZyaMZKu3cT6ujSCKotISDMc3Aj22YgraMcDrKthJDYyAD6NGmoxIENWE62XiET53SwV2l3ZMWT9S/P1IQxgxF5JICbN8semPxP6+Z2O5VmDIZJ5ZKMh3UTTi2Co/Pxx2mKbF86AgQzdyumPRBA7GupLkpxrqKjAuYonBRZUauomCxkGVSOy8FQSm4u8iXr5+mZWXQMTpBBRSgS1RGt6iCqoggiV7QK3rznr1378P7nEZXvFnBR2gO3tcvRqqdjQ==</latexit><latexit sha1_base64="2QtFLFbiPIjQhyO4MxS4aXLrHLw=">AAACD3icbVDLSgMxFM34rPVVdekmWIS6aJkRQZcVNy4r2Ae2Q7mTpm1oHkOSEcpQ8BcEt/ob7sStn+Bf+Ammj4VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdnKbu/s7u3nDg5rRiWa0CpRXOlGBIZyJmnVMstpI9YURMRpPRrcjP36I9WGKXlvhzENBfQk6zIC1kkPhaDYAh734aydy/slfwK8TIIZyaMZKu3cT6ujSCKotISDMc3Aj22YgraMcDrKthJDYyAD6NGmoxIENWE62XiET53SwV2l3ZMWT9S/P1IQxgxF5JICbN8semPxP6+Z2O5VmDIZJ5ZKMh3UTTi2Co/Pxx2mKbF86AgQzdyumPRBA7GupLkpxrqKjAuYonBRZUauomCxkGVSOy8FQSm4u8iXr5+mZWXQMTpBBRSgS1RGt6iCqoggiV7QK3rznr1378P7nEZXvFnBR2gO3tcvRqqdjQ==</latexit><latexit sha1_base64="2QtFLFbiPIjQhyO4MxS4aXLrHLw=">AAACD3icbVDLSgMxFM34rPVVdekmWIS6aJkRQZcVNy4r2Ae2Q7mTpm1oHkOSEcpQ8BcEt/ob7sStn+Bf+Ammj4VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdnKbu/s7u3nDg5rRiWa0CpRXOlGBIZyJmnVMstpI9YURMRpPRrcjP36I9WGKXlvhzENBfQk6zIC1kkPhaDYAh734aydy/slfwK8TIIZyaMZKu3cT6ujSCKotISDMc3Aj22YgraMcDrKthJDYyAD6NGmoxIENWE62XiET53SwV2l3ZMWT9S/P1IQxgxF5JICbN8semPxP6+Z2O5VmDIZJ5ZKMh3UTTi2Co/Pxx2mKbF86AgQzdyumPRBA7GupLkpxrqKjAuYonBRZUauomCxkGVSOy8FQSm4u8iXr5+mZWXQMTpBBRSgS1RGt6iCqoggiV7QK3rznr1378P7nEZXvFnBR2gO3tcvRqqdjQ==</latexit>

CVaR↵(✓;P ) := EP [`(✓;Z) | `(✓;Z) � P�1(1� ↵)]

= inf⌘

⇢1

↵EP (`(✓;Z)� ⌘)+ + ⌘

<latexit sha1_base64="zHqWiDR0S/4n68GhQbtMpsuXI6E=">AAAC63icbVJdixMxFM2MX2v9qvroS7AoLUvLRARFEVYWQXwaxXYXm3HIZDKdsJkPkjtiDQP+B9/EV3+U7/4Af4KZtiy29ULgcO7NuTfnJqmVNBAEvzz/wsVLl68cXO1du37j5q3+7TszUzWaiymvVKVPE2aEkqWYggQlTmstWJEocZKcHXf5k09CG1mV72FZi6hgi1JmkjNwVNz/TYuk+myPZ+xdG1uqRWopU3XO2nZIIRfAnuNwhHsP8bMXmBYM8iSxr9o4nGMqlDqv+TByWZnukwuBw492TJxcJ07GG/lRhCntdJ2sLDPX211pMVUiA2rxepJMM25Jez7S1gTD3V5j3GmM4kN8uEJORC5yoG3cHwSTYBV4H5ANGKBNhHH/D00r3hSiBK6YMXMS1BBZpkFyJdoebYyoGT9jCzF3sGSFMJFdbaPFDxyT4qzS7pSAV+y/NywrjFkWiavsXmN2cx35v9y8gexpZGVZNyBKvm6UNQpDhbvV4lRqwUEtHWBcSzcr5jlzFoL7AFtdDLj1G1dgxoUrrUxnEdk1ZB/MHk0ImZC3jwdHL7+uzTpA99B9NEQEPUFH6DUK0RRx741Xe0vvi1/43/zv/o91qe9tDL6LtsL/+RdjSehr</latexit><latexit sha1_base64="zHqWiDR0S/4n68GhQbtMpsuXI6E=">AAAC63icbVJdixMxFM2MX2v9qvroS7AoLUvLRARFEVYWQXwaxXYXm3HIZDKdsJkPkjtiDQP+B9/EV3+U7/4Af4KZtiy29ULgcO7NuTfnJqmVNBAEvzz/wsVLl68cXO1du37j5q3+7TszUzWaiymvVKVPE2aEkqWYggQlTmstWJEocZKcHXf5k09CG1mV72FZi6hgi1JmkjNwVNz/TYuk+myPZ+xdG1uqRWopU3XO2nZIIRfAnuNwhHsP8bMXmBYM8iSxr9o4nGMqlDqv+TByWZnukwuBw492TJxcJ07GG/lRhCntdJ2sLDPX211pMVUiA2rxepJMM25Jez7S1gTD3V5j3GmM4kN8uEJORC5yoG3cHwSTYBV4H5ANGKBNhHH/D00r3hSiBK6YMXMS1BBZpkFyJdoebYyoGT9jCzF3sGSFMJFdbaPFDxyT4qzS7pSAV+y/NywrjFkWiavsXmN2cx35v9y8gexpZGVZNyBKvm6UNQpDhbvV4lRqwUEtHWBcSzcr5jlzFoL7AFtdDLj1G1dgxoUrrUxnEdk1ZB/MHk0ImZC3jwdHL7+uzTpA99B9NEQEPUFH6DUK0RRx741Xe0vvi1/43/zv/o91qe9tDL6LtsL/+RdjSehr</latexit><latexit sha1_base64="zHqWiDR0S/4n68GhQbtMpsuXI6E=">AAAC63icbVJdixMxFM2MX2v9qvroS7AoLUvLRARFEVYWQXwaxXYXm3HIZDKdsJkPkjtiDQP+B9/EV3+U7/4Af4KZtiy29ULgcO7NuTfnJqmVNBAEvzz/wsVLl68cXO1du37j5q3+7TszUzWaiymvVKVPE2aEkqWYggQlTmstWJEocZKcHXf5k09CG1mV72FZi6hgi1JmkjNwVNz/TYuk+myPZ+xdG1uqRWopU3XO2nZIIRfAnuNwhHsP8bMXmBYM8iSxr9o4nGMqlDqv+TByWZnukwuBw492TJxcJ07GG/lRhCntdJ2sLDPX211pMVUiA2rxepJMM25Jez7S1gTD3V5j3GmM4kN8uEJORC5yoG3cHwSTYBV4H5ANGKBNhHH/D00r3hSiBK6YMXMS1BBZpkFyJdoebYyoGT9jCzF3sGSFMJFdbaPFDxyT4qzS7pSAV+y/NywrjFkWiavsXmN2cx35v9y8gexpZGVZNyBKvm6UNQpDhbvV4lRqwUEtHWBcSzcr5jlzFoL7AFtdDLj1G1dgxoUrrUxnEdk1ZB/MHk0ImZC3jwdHL7+uzTpA99B9NEQEPUFH6DUK0RRx741Xe0vvi1/43/zv/o91qe9tDL6LtsL/+RdjSehr</latexit><latexit sha1_base64="zHqWiDR0S/4n68GhQbtMpsuXI6E=">AAAC63icbVJdixMxFM2MX2v9qvroS7AoLUvLRARFEVYWQXwaxXYXm3HIZDKdsJkPkjtiDQP+B9/EV3+U7/4Af4KZtiy29ULgcO7NuTfnJqmVNBAEvzz/wsVLl68cXO1du37j5q3+7TszUzWaiymvVKVPE2aEkqWYggQlTmstWJEocZKcHXf5k09CG1mV72FZi6hgi1JmkjNwVNz/TYuk+myPZ+xdG1uqRWopU3XO2nZIIRfAnuNwhHsP8bMXmBYM8iSxr9o4nGMqlDqv+TByWZnukwuBw492TJxcJ07GG/lRhCntdJ2sLDPX211pMVUiA2rxepJMM25Jez7S1gTD3V5j3GmM4kN8uEJORC5yoG3cHwSTYBV4H5ANGKBNhHH/D00r3hSiBK6YMXMS1BBZpkFyJdoebYyoGT9jCzF3sGSFMJFdbaPFDxyT4qzS7pSAV+y/NywrjFkWiavsXmN2cx35v9y8gexpZGVZNyBKvm6UNQpDhbvV4lRqwUEtHWBcSzcr5jlzFoL7AFtdDLj1G1dgxoUrrUxnEdk1ZB/MHk0ImZC3jwdHL7+uzTpA99B9NEQEPUFH6DUK0RRx741Xe0vvi1/43/zv/o91qe9tDL6LtsL/+RdjSehr</latexit>

P�1(1� ↵)<latexit sha1_base64="SRnJFFd5405hMiT7iCrCPgDMR3o=">AAACHXicbVDLSsNAFJ3UV62vqODGTbAIddGSEUGXRTcuK9gHNLFMppN26EwmzEyEEvMtglv9DXfiVvwLP8Fpm4VtPXDhcM653MsJYkaVdt1vq7Cyura+UdwsbW3v7O7Z+wctJRKJSRMLJmQnQIowGpGmppqRTiwJ4gEj7WB0M/Hbj0QqKqJ7PY6Jz9EgoiHFSBupZx81HtIqzCqeJP0UVj3E4iHKznp22a25UzjLBOakDHI0evaP1xc44STSmCGlutCNtZ8iqSlmJCt5iSIxwiM0IF1DI8SJ8tPp/5lzapS+EwppJtLOVP27kSKu1JgHJsmRHqpFbyL+53UTHV75KY3iRJMIzw6FCXO0cCZlOH0qCdZsbAjCkppfHTxEEmFtKpu7orQpTJmAqnITFSozFcHFQpZJ67wGYQ3eXZTr13lZRXAMTkAFQHAJ6uAWNEATYPAEXsAreLOerXfrw/qcRQtWvnMI5mB9/QIVUKJK</latexit><latexit sha1_base64="SRnJFFd5405hMiT7iCrCPgDMR3o=">AAACHXicbVDLSsNAFJ3UV62vqODGTbAIddGSEUGXRTcuK9gHNLFMppN26EwmzEyEEvMtglv9DXfiVvwLP8Fpm4VtPXDhcM653MsJYkaVdt1vq7Cyura+UdwsbW3v7O7Z+wctJRKJSRMLJmQnQIowGpGmppqRTiwJ4gEj7WB0M/Hbj0QqKqJ7PY6Jz9EgoiHFSBupZx81HtIqzCqeJP0UVj3E4iHKznp22a25UzjLBOakDHI0evaP1xc44STSmCGlutCNtZ8iqSlmJCt5iSIxwiM0IF1DI8SJ8tPp/5lzapS+EwppJtLOVP27kSKu1JgHJsmRHqpFbyL+53UTHV75KY3iRJMIzw6FCXO0cCZlOH0qCdZsbAjCkppfHTxEEmFtKpu7orQpTJmAqnITFSozFcHFQpZJ67wGYQ3eXZTr13lZRXAMTkAFQHAJ6uAWNEATYPAEXsAreLOerXfrw/qcRQtWvnMI5mB9/QIVUKJK</latexit><latexit sha1_base64="SRnJFFd5405hMiT7iCrCPgDMR3o=">AAACHXicbVDLSsNAFJ3UV62vqODGTbAIddGSEUGXRTcuK9gHNLFMppN26EwmzEyEEvMtglv9DXfiVvwLP8Fpm4VtPXDhcM653MsJYkaVdt1vq7Cyura+UdwsbW3v7O7Z+wctJRKJSRMLJmQnQIowGpGmppqRTiwJ4gEj7WB0M/Hbj0QqKqJ7PY6Jz9EgoiHFSBupZx81HtIqzCqeJP0UVj3E4iHKznp22a25UzjLBOakDHI0evaP1xc44STSmCGlutCNtZ8iqSlmJCt5iSIxwiM0IF1DI8SJ8tPp/5lzapS+EwppJtLOVP27kSKu1JgHJsmRHqpFbyL+53UTHV75KY3iRJMIzw6FCXO0cCZlOH0qCdZsbAjCkppfHTxEEmFtKpu7orQpTJmAqnITFSozFcHFQpZJ67wGYQ3eXZTr13lZRXAMTkAFQHAJ6uAWNEATYPAEXsAreLOerXfrw/qcRQtWvnMI5mB9/QIVUKJK</latexit><latexit sha1_base64="SRnJFFd5405hMiT7iCrCPgDMR3o=">AAACHXicbVDLSsNAFJ3UV62vqODGTbAIddGSEUGXRTcuK9gHNLFMppN26EwmzEyEEvMtglv9DXfiVvwLP8Fpm4VtPXDhcM653MsJYkaVdt1vq7Cyura+UdwsbW3v7O7Z+wctJRKJSRMLJmQnQIowGpGmppqRTiwJ4gEj7WB0M/Hbj0QqKqJ7PY6Jz9EgoiHFSBupZx81HtIqzCqeJP0UVj3E4iHKznp22a25UzjLBOakDHI0evaP1xc44STSmCGlutCNtZ8iqSlmJCt5iSIxwiM0IF1DI8SJ8tPp/5lzapS+EwppJtLOVP27kSKu1JgHJsmRHqpFbyL+53UTHV75KY3iRJMIzw6FCXO0cCZlOH0qCdZsbAjCkppfHTxEEmFtKpu7orQpTJmAqnITFSozFcHFQpZJ67wGYQ3eXZTr13lZRXAMTkAFQHAJ6uAWNEATYPAEXsAreLOerXfrw/qcRQtWvnMI5mB9/QIVUKJK</latexit>

Page 22: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Random Minority Proportion• Instead of a fixed minority proportion, consider random

minority proportion

• Consider worst-case formulation over random minority proportions

Worst-case over all subpopulations larger than A<latexit sha1_base64="0Wftv6WskB0KnGvPqlQ6hu6q/gI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scyIoMsWNy4r2Ae0Q8mkt21oHkOSEcpQ8A8Et/ob7sSt3+Bf+Alm2i5s64HA4ZwT7r0nijkz1ve/vdza+sbmVn67sLO7t39QPDxqGJVoCnWquNKtiBjgTELdMsuhFWsgIuLQjEa3md98BG2Ykg92HEMoyECyPqPEOqnViXgCaXXSLZb8sj8FXiXBnJTQHLVu8afTUzQRIC3lxJh24Mc2TIm2jHKYFDqJgZjQERlA21FJBJgwne47wWdO6eG+0u5Ji6fq3x8pEcaMReSSgtihWfYy8T+vndj+TZgyGScWJJ0N6iccW4Wz43GPaaCWjx0hVDO3K6ZDogm1rqKFKca6gowLmAvhospkFQXLhaySxmU5cPz+qlSpPs3KyqMTdIrOUYCuUQXdoRqqI4o4ekGv6M179t69D+9zFs1584KP0QK8r19Cy52k</latexit><latexit sha1_base64="0Wftv6WskB0KnGvPqlQ6hu6q/gI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scyIoMsWNy4r2Ae0Q8mkt21oHkOSEcpQ8A8Et/ob7sSt3+Bf+Alm2i5s64HA4ZwT7r0nijkz1ve/vdza+sbmVn67sLO7t39QPDxqGJVoCnWquNKtiBjgTELdMsuhFWsgIuLQjEa3md98BG2Ykg92HEMoyECyPqPEOqnViXgCaXXSLZb8sj8FXiXBnJTQHLVu8afTUzQRIC3lxJh24Mc2TIm2jHKYFDqJgZjQERlA21FJBJgwne47wWdO6eG+0u5Ji6fq3x8pEcaMReSSgtihWfYy8T+vndj+TZgyGScWJJ0N6iccW4Wz43GPaaCWjx0hVDO3K6ZDogm1rqKFKca6gowLmAvhospkFQXLhaySxmU5cPz+qlSpPs3KyqMTdIrOUYCuUQXdoRqqI4o4ekGv6M179t69D+9zFs1584KP0QK8r19Cy52k</latexit><latexit sha1_base64="0Wftv6WskB0KnGvPqlQ6hu6q/gI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scyIoMsWNy4r2Ae0Q8mkt21oHkOSEcpQ8A8Et/ob7sSt3+Bf+Alm2i5s64HA4ZwT7r0nijkz1ve/vdza+sbmVn67sLO7t39QPDxqGJVoCnWquNKtiBjgTELdMsuhFWsgIuLQjEa3md98BG2Ykg92HEMoyECyPqPEOqnViXgCaXXSLZb8sj8FXiXBnJTQHLVu8afTUzQRIC3lxJh24Mc2TIm2jHKYFDqJgZjQERlA21FJBJgwne47wWdO6eG+0u5Ji6fq3x8pEcaMReSSgtihWfYy8T+vndj+TZgyGScWJJ0N6iccW4Wz43GPaaCWjx0hVDO3K6ZDogm1rqKFKca6gowLmAvhospkFQXLhaySxmU5cPz+qlSpPs3KyqMTdIrOUYCuUQXdoRqqI4o4ekGv6M179t69D+9zFs1584KP0QK8r19Cy52k</latexit><latexit sha1_base64="0Wftv6WskB0KnGvPqlQ6hu6q/gI=">AAACDXicbVDLSgMxFM3UV62vqks3wSK4scyIoMsWNy4r2Ae0Q8mkt21oHkOSEcpQ8A8Et/ob7sSt3+Bf+Alm2i5s64HA4ZwT7r0nijkz1ve/vdza+sbmVn67sLO7t39QPDxqGJVoCnWquNKtiBjgTELdMsuhFWsgIuLQjEa3md98BG2Ykg92HEMoyECyPqPEOqnViXgCaXXSLZb8sj8FXiXBnJTQHLVu8afTUzQRIC3lxJh24Mc2TIm2jHKYFDqJgZjQERlA21FJBJgwne47wWdO6eG+0u5Ji6fq3x8pEcaMReSSgtihWfYy8T+vndj+TZgyGScWJJ0N6iccW4Wz43GPaaCWjx0hVDO3K6ZDogm1rqKFKca6gowLmAvhospkFQXLhaySxmU5cPz+qlSpPs3KyqMTdIrOUYCuUQXdoRqqI4o4ekGv6M179t69D+9zFs1584KP0QK8r19Cy52k</latexit>

A ⇠ PA<latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit>

A ⇠ PA<latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit><latexit sha1_base64="xqO/w9fdk6kTMcRxNDYaTyM9yko=">AAACGHicbZDLSgMxGIUz9VbrbapLN8EiuLHMiKDLFjcuK9gLtMOQSTNtaC5DklHKUPA5BLf6Gu7ErTvfwkcw03ZhW38IHM454f/5ooRRbTzv2ymsrW9sbhW3Szu7e/sHbvmwpWWqMGliyaTqREgTRgVpGmoY6SSKIB4x0o5GN3nefiBKUynuzTghAUcDQWOKkbFW6JZ7EUtJVoc9TTlshPVJ6Fa8qjcduCr8uaiA+TRC96fXlzjlRBjMkNZd30tMkCFlKGZkUuqlmiQIj9CAdK0UiBMdZNPTJ/DUOn0YS2WfMHDq/v2RIa71mEe2yZEZ6uUsN//LuqmJr4OMiiQ1RODZojhl0EiYc4B9qgg2bGwFworaWyEeIoWwsbQWtmhjWWlb0OfcVqXOEfnLQFZF66LqW313WanVn2awiuAYnIAz4IMrUAO3oAGaAINH8AJewZvz7Lw7H87nrFpw5oCPwMI4X7/TR6EE</latexit>

supPA2PA

EA⇠PACVaRA(✓;P )<latexit sha1_base64="oCFfvoVUIXblWBkM7WgWV++b0X8=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLYMBNQhRctuJMAlNDU119J1NMPZqq28GhqP/nX/A3COJOl9Y8FJN4oeBwzrnc4qtbJT3m+ddedu36jZu3tm5v37l77/5Of/fByNvOCRgKq6w7rbkHJQ0MUaKC09YB17WCk3p+vMxPzsF5ac0nXLQw0fzMyKkUHJNV9Wvmu7YKrFYdhLI6okwayrQIZayOYqTs3Z8wRV5qWq7thru5gyYwXdvP4XjEP8a/xRj3GM4A+RtaPqex6g/y/Xw19KooNmJANlNW/R+ssaLTYFAo7v24yFucBO5QCgVxm3UeWi7m/AzGSRquwU/CikWkz5LT0Kl16RmkK/ffjcC19wtdp6bmOPOXs6X5v2zc4fRgEqRpOwQj1oemnaJo6RIsbaQDgWqRBBdOpr9SMeOOC0z4L1zxmOD7VPAvdKpav0RUXAZyVYxe7hdJf3g1OHy7gbVFnpCnZI8U5DU5JO9JSYZEkC/kO/lJfvW+ZbvZo+zxupr1NjsPyYXJBr8BjGu+FA==</latexit><latexit sha1_base64="oCFfvoVUIXblWBkM7WgWV++b0X8=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLYMBNQhRctuJMAlNDU119J1NMPZqq28GhqP/nX/A3COJOl9Y8FJN4oeBwzrnc4qtbJT3m+ddedu36jZu3tm5v37l77/5Of/fByNvOCRgKq6w7rbkHJQ0MUaKC09YB17WCk3p+vMxPzsF5ac0nXLQw0fzMyKkUHJNV9Wvmu7YKrFYdhLI6okwayrQIZayOYqTs3Z8wRV5qWq7thru5gyYwXdvP4XjEP8a/xRj3GM4A+RtaPqex6g/y/Xw19KooNmJANlNW/R+ssaLTYFAo7v24yFucBO5QCgVxm3UeWi7m/AzGSRquwU/CikWkz5LT0Kl16RmkK/ffjcC19wtdp6bmOPOXs6X5v2zc4fRgEqRpOwQj1oemnaJo6RIsbaQDgWqRBBdOpr9SMeOOC0z4L1zxmOD7VPAvdKpav0RUXAZyVYxe7hdJf3g1OHy7gbVFnpCnZI8U5DU5JO9JSYZEkC/kO/lJfvW+ZbvZo+zxupr1NjsPyYXJBr8BjGu+FA==</latexit><latexit sha1_base64="oCFfvoVUIXblWBkM7WgWV++b0X8=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLYMBNQhRctuJMAlNDU119J1NMPZqq28GhqP/nX/A3COJOl9Y8FJN4oeBwzrnc4qtbJT3m+ddedu36jZu3tm5v37l77/5Of/fByNvOCRgKq6w7rbkHJQ0MUaKC09YB17WCk3p+vMxPzsF5ac0nXLQw0fzMyKkUHJNV9Wvmu7YKrFYdhLI6okwayrQIZayOYqTs3Z8wRV5qWq7thru5gyYwXdvP4XjEP8a/xRj3GM4A+RtaPqex6g/y/Xw19KooNmJANlNW/R+ssaLTYFAo7v24yFucBO5QCgVxm3UeWi7m/AzGSRquwU/CikWkz5LT0Kl16RmkK/ffjcC19wtdp6bmOPOXs6X5v2zc4fRgEqRpOwQj1oemnaJo6RIsbaQDgWqRBBdOpr9SMeOOC0z4L1zxmOD7VPAvdKpav0RUXAZyVYxe7hdJf3g1OHy7gbVFnpCnZI8U5DU5JO9JSYZEkC/kO/lJfvW+ZbvZo+zxupr1NjsPyYXJBr8BjGu+FA==</latexit><latexit sha1_base64="oCFfvoVUIXblWBkM7WgWV++b0X8=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLYMBNQhRctuJMAlNDU119J1NMPZqq28GhqP/nX/A3COJOl9Y8FJN4oeBwzrnc4qtbJT3m+ddedu36jZu3tm5v37l77/5Of/fByNvOCRgKq6w7rbkHJQ0MUaKC09YB17WCk3p+vMxPzsF5ac0nXLQw0fzMyKkUHJNV9Wvmu7YKrFYdhLI6okwayrQIZayOYqTs3Z8wRV5qWq7thru5gyYwXdvP4XjEP8a/xRj3GM4A+RtaPqex6g/y/Xw19KooNmJANlNW/R+ssaLTYFAo7v24yFucBO5QCgVxm3UeWi7m/AzGSRquwU/CikWkz5LT0Kl16RmkK/ffjcC19wtdp6bmOPOXs6X5v2zc4fRgEqRpOwQj1oemnaJo6RIsbaQDgWqRBBdOpr9SMeOOC0z4L1zxmOD7VPAvdKpav0RUXAZyVYxe7hdJf3g1OHy7gbVFnpCnZI8U5DU5JO9JSYZEkC/kO/lJfvW+ZbvZo+zxupr1NjsPyYXJBr8BjGu+FA==</latexit>

EA⇠PACVaRA(✓;P )<latexit sha1_base64="hGT4KqywHn4aSpVTZpdckA6dvQY=">AAACSnicbVBdSxtBFJ2NWj9bU/vYl6GhoA+VXREUfFHSQh+jmChkw3J39sYMmdlZZu6KYdkf1P8i+Fr/Relb6YuTGKRqDwwczjmXe+ekhZKOwvA+aCwsLr1ZXlldW994+26z+X6r50xpBXaFUcZepuBQyRy7JEnhZWERdKrwIh23p/7FNVonTX5OkwIHGq5yOZQCyEtJsx1/S6o4VSVWJzx2UvNOclLXPM7Aji1mVaxTc1O1e3BWPwXrejumERIc8c4Or5NmK9wNZ+CvSTQnLTZHJ2n+ijMjSo05CQXO9aOwoEEFlqRQWK/FpcMCxBiusO9pDhrdoJp9tuafvZLxobH+5cRn6r8TFWjnJjr1SQ00ci+9qfg/r1/S8HBQybwoCXPxuGhYKk6GT5vjmbQoSE08AWGlv5WLEVgQ5Pt9tsWRb9f5gPuifdS4aUXRy0Jek97ebuT56X7r+Ou8rBX2kX1i2yxiB+yYfWcd1mWC/WB37Ce7D26D38Gf4O9jtBHMZz6wZ2gsPgCqYrQ9</latexit><latexit sha1_base64="hGT4KqywHn4aSpVTZpdckA6dvQY=">AAACSnicbVBdSxtBFJ2NWj9bU/vYl6GhoA+VXREUfFHSQh+jmChkw3J39sYMmdlZZu6KYdkf1P8i+Fr/Relb6YuTGKRqDwwczjmXe+ekhZKOwvA+aCwsLr1ZXlldW994+26z+X6r50xpBXaFUcZepuBQyRy7JEnhZWERdKrwIh23p/7FNVonTX5OkwIHGq5yOZQCyEtJsx1/S6o4VSVWJzx2UvNOclLXPM7Aji1mVaxTc1O1e3BWPwXrejumERIc8c4Or5NmK9wNZ+CvSTQnLTZHJ2n+ijMjSo05CQXO9aOwoEEFlqRQWK/FpcMCxBiusO9pDhrdoJp9tuafvZLxobH+5cRn6r8TFWjnJjr1SQ00ci+9qfg/r1/S8HBQybwoCXPxuGhYKk6GT5vjmbQoSE08AWGlv5WLEVgQ5Pt9tsWRb9f5gPuifdS4aUXRy0Jek97ebuT56X7r+Ou8rBX2kX1i2yxiB+yYfWcd1mWC/WB37Ce7D26D38Gf4O9jtBHMZz6wZ2gsPgCqYrQ9</latexit><latexit sha1_base64="hGT4KqywHn4aSpVTZpdckA6dvQY=">AAACSnicbVBdSxtBFJ2NWj9bU/vYl6GhoA+VXREUfFHSQh+jmChkw3J39sYMmdlZZu6KYdkf1P8i+Fr/Relb6YuTGKRqDwwczjmXe+ekhZKOwvA+aCwsLr1ZXlldW994+26z+X6r50xpBXaFUcZepuBQyRy7JEnhZWERdKrwIh23p/7FNVonTX5OkwIHGq5yOZQCyEtJsx1/S6o4VSVWJzx2UvNOclLXPM7Aji1mVaxTc1O1e3BWPwXrejumERIc8c4Or5NmK9wNZ+CvSTQnLTZHJ2n+ijMjSo05CQXO9aOwoEEFlqRQWK/FpcMCxBiusO9pDhrdoJp9tuafvZLxobH+5cRn6r8TFWjnJjr1SQ00ci+9qfg/r1/S8HBQybwoCXPxuGhYKk6GT5vjmbQoSE08AWGlv5WLEVgQ5Pt9tsWRb9f5gPuifdS4aUXRy0Jek97ebuT56X7r+Ou8rBX2kX1i2yxiB+yYfWcd1mWC/WB37Ce7D26D38Gf4O9jtBHMZz6wZ2gsPgCqYrQ9</latexit><latexit sha1_base64="hGT4KqywHn4aSpVTZpdckA6dvQY=">AAACSnicbVBdSxtBFJ2NWj9bU/vYl6GhoA+VXREUfFHSQh+jmChkw3J39sYMmdlZZu6KYdkf1P8i+Fr/Relb6YuTGKRqDwwczjmXe+ekhZKOwvA+aCwsLr1ZXlldW994+26z+X6r50xpBXaFUcZepuBQyRy7JEnhZWERdKrwIh23p/7FNVonTX5OkwIHGq5yOZQCyEtJsx1/S6o4VSVWJzx2UvNOclLXPM7Aji1mVaxTc1O1e3BWPwXrejumERIc8c4Or5NmK9wNZ+CvSTQnLTZHJ2n+ijMjSo05CQXO9aOwoEEFlqRQWK/FpcMCxBiusO9pDhrdoJp9tuafvZLxobH+5cRn6r8TFWjnJjr1SQ00ci+9qfg/r1/S8HBQybwoCXPxuGhYKk6GT5vjmbQoSE08AWGlv5WLEVgQ5Pt9tsWRb9f5gPuifdS4aUXRy0Jek97ebuT56X7r+Ou8rBX2kX1i2yxiB+yYfWcd1mWC/WB37Ce7D26D38Gf4O9jtBHMZz6wZ2gsPgCqYrQ9</latexit>

Worst-case over probability on minority proportionPA<latexit sha1_base64="WtS+gABWWJjalLHH0luZyCS5HkI=">AAACD3icbVDLSgMxFM34rPVVdekmWAQ3lhkRdNnixmUF+8B2KJn0tg3NY0gyQhkK/oLgVn/Dnbj1E/wLP8FM24VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdvKb+/s7u0XDg7rRiWaQo0qrnQzIgY4k1CzzHJoxhqIiDg0ouFN5jceQRum5L0dxRAK0pesxyixTnpoRzyBtNqpjDuFol/yJ8DLJJiRIpqh2in8tLuKJgKkpZwY0wr82IYp0ZZRDuN8OzEQEzokfWg5KokAE6aTjcf41Cld3FPaPWnxRP37IyXCmJGIXFIQOzCLXib+57US27sOUybjxIKk00G9hGOrcHY+7jIN1PKRI4Rq5nbFdEA0odaVNDfFWFeRcQFzLlxUmayiYLGQZVK/KAWO310Wy5WnaVk5dIxO0BkK0BUqo1tURTVEkUQv6BW9ec/eu/fhfU6jK96s4CM0B+/rF7Fmnmc=</latexit><latexit sha1_base64="WtS+gABWWJjalLHH0luZyCS5HkI=">AAACD3icbVDLSgMxFM34rPVVdekmWAQ3lhkRdNnixmUF+8B2KJn0tg3NY0gyQhkK/oLgVn/Dnbj1E/wLP8FM24VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdvKb+/s7u0XDg7rRiWaQo0qrnQzIgY4k1CzzHJoxhqIiDg0ouFN5jceQRum5L0dxRAK0pesxyixTnpoRzyBtNqpjDuFol/yJ8DLJJiRIpqh2in8tLuKJgKkpZwY0wr82IYp0ZZRDuN8OzEQEzokfWg5KokAE6aTjcf41Cld3FPaPWnxRP37IyXCmJGIXFIQOzCLXib+57US27sOUybjxIKk00G9hGOrcHY+7jIN1PKRI4Rq5nbFdEA0odaVNDfFWFeRcQFzLlxUmayiYLGQZVK/KAWO310Wy5WnaVk5dIxO0BkK0BUqo1tURTVEkUQv6BW9ec/eu/fhfU6jK96s4CM0B+/rF7Fmnmc=</latexit><latexit sha1_base64="WtS+gABWWJjalLHH0luZyCS5HkI=">AAACD3icbVDLSgMxFM34rPVVdekmWAQ3lhkRdNnixmUF+8B2KJn0tg3NY0gyQhkK/oLgVn/Dnbj1E/wLP8FM24VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdvKb+/s7u0XDg7rRiWaQo0qrnQzIgY4k1CzzHJoxhqIiDg0ouFN5jceQRum5L0dxRAK0pesxyixTnpoRzyBtNqpjDuFol/yJ8DLJJiRIpqh2in8tLuKJgKkpZwY0wr82IYp0ZZRDuN8OzEQEzokfWg5KokAE6aTjcf41Cld3FPaPWnxRP37IyXCmJGIXFIQOzCLXib+57US27sOUybjxIKk00G9hGOrcHY+7jIN1PKRI4Rq5nbFdEA0odaVNDfFWFeRcQFzLlxUmayiYLGQZVK/KAWO310Wy5WnaVk5dIxO0BkK0BUqo1tURTVEkUQv6BW9ec/eu/fhfU6jK96s4CM0B+/rF7Fmnmc=</latexit><latexit sha1_base64="WtS+gABWWJjalLHH0luZyCS5HkI=">AAACD3icbVDLSgMxFM34rPVVdekmWAQ3lhkRdNnixmUF+8B2KJn0tg3NY0gyQhkK/oLgVn/Dnbj1E/wLP8FM24VtPRA4nHPCvfdEMWfG+v63t7K6tr6xmdvKb+/s7u0XDg7rRiWaQo0qrnQzIgY4k1CzzHJoxhqIiDg0ouFN5jceQRum5L0dxRAK0pesxyixTnpoRzyBtNqpjDuFol/yJ8DLJJiRIpqh2in8tLuKJgKkpZwY0wr82IYp0ZZRDuN8OzEQEzokfWg5KokAE6aTjcf41Cld3FPaPWnxRP37IyXCmJGIXFIQOzCLXib+57US27sOUybjxIKk00G9hGOrcHY+7jIN1PKRI4Rq5nbFdEA0odaVNDfFWFeRcQFzLlxUmayiYLGQZVK/KAWO310Wy5WnaVk5dIxO0BkK0BUqo1tURTVEkUQv6BW9ec/eu/fhfU6jK96s4CM0B+/rF7Fmnmc=</latexit>

• 1-1 correspondence with coherent risk measures [Kusuoka ’01]

Page 23: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Operationalization

Average-case

Worst-off individual

interpolate via PA<latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit>

EPobs [`(✓;Z)]<latexit sha1_base64="JEft4SkhO4LHZvpVBCR1OiyUfqA=">AAACNnicbZDLShxBFIar1XiNOpqlm8JBMItotwQiuFEkkOUEnFEy3QzVNWecYqq6mjqnhaFp8CV8F8GteYhsshO38wipuSy85EDBX//5D6fqS3OtkMLwTzA3v/BhcWl5ZXXt4/rGZm1ru4W2cBKa0mrrrlKBoFUGTVKk4Sp3IEyq4TIdnI/7lzfgUNnsgoY5JEZcZ6qnpCBvdWqH8fdOGXeFG+SFyzWUDX91htsUq6pqx9oi7sfUBxIn/NfnpFOrhwfhpPh7Ec1Enc2q0amN4q6VhYGMpBaI7SjMKSmFIyU1VKtxgZALORDX0PYyEwYwKScfq/ied7q8Z50/GfGJ+3KiFAZxaFKfNIL6+LY3Nv/XaxfUO05KleUFQSani3qF5mT5mBLvKgeS9NALIZ3yb+WyL5yQ5Fm+2oLkSaIP4BfjoxYrjyh6C+S9aB0dRF7//Fo/PbudwlpmO2yX7bOIfWOn7AdrsCaT7I49sEf2O7gP/gZPwfM0OhfMAH9iryoY/QNWqa43</latexit><latexit sha1_base64="JEft4SkhO4LHZvpVBCR1OiyUfqA=">AAACNnicbZDLShxBFIar1XiNOpqlm8JBMItotwQiuFEkkOUEnFEy3QzVNWecYqq6mjqnhaFp8CV8F8GteYhsshO38wipuSy85EDBX//5D6fqS3OtkMLwTzA3v/BhcWl5ZXXt4/rGZm1ru4W2cBKa0mrrrlKBoFUGTVKk4Sp3IEyq4TIdnI/7lzfgUNnsgoY5JEZcZ6qnpCBvdWqH8fdOGXeFG+SFyzWUDX91htsUq6pqx9oi7sfUBxIn/NfnpFOrhwfhpPh7Ec1Enc2q0amN4q6VhYGMpBaI7SjMKSmFIyU1VKtxgZALORDX0PYyEwYwKScfq/ied7q8Z50/GfGJ+3KiFAZxaFKfNIL6+LY3Nv/XaxfUO05KleUFQSani3qF5mT5mBLvKgeS9NALIZ3yb+WyL5yQ5Fm+2oLkSaIP4BfjoxYrjyh6C+S9aB0dRF7//Fo/PbudwlpmO2yX7bOIfWOn7AdrsCaT7I49sEf2O7gP/gZPwfM0OhfMAH9iryoY/QNWqa43</latexit><latexit sha1_base64="JEft4SkhO4LHZvpVBCR1OiyUfqA=">AAACNnicbZDLShxBFIar1XiNOpqlm8JBMItotwQiuFEkkOUEnFEy3QzVNWecYqq6mjqnhaFp8CV8F8GteYhsshO38wipuSy85EDBX//5D6fqS3OtkMLwTzA3v/BhcWl5ZXXt4/rGZm1ru4W2cBKa0mrrrlKBoFUGTVKk4Sp3IEyq4TIdnI/7lzfgUNnsgoY5JEZcZ6qnpCBvdWqH8fdOGXeFG+SFyzWUDX91htsUq6pqx9oi7sfUBxIn/NfnpFOrhwfhpPh7Ec1Enc2q0amN4q6VhYGMpBaI7SjMKSmFIyU1VKtxgZALORDX0PYyEwYwKScfq/ied7q8Z50/GfGJ+3KiFAZxaFKfNIL6+LY3Nv/XaxfUO05KleUFQSani3qF5mT5mBLvKgeS9NALIZ3yb+WyL5yQ5Fm+2oLkSaIP4BfjoxYrjyh6C+S9aB0dRF7//Fo/PbudwlpmO2yX7bOIfWOn7AdrsCaT7I49sEf2O7gP/gZPwfM0OhfMAH9iryoY/QNWqa43</latexit><latexit sha1_base64="JEft4SkhO4LHZvpVBCR1OiyUfqA=">AAACNnicbZDLShxBFIar1XiNOpqlm8JBMItotwQiuFEkkOUEnFEy3QzVNWecYqq6mjqnhaFp8CV8F8GteYhsshO38wipuSy85EDBX//5D6fqS3OtkMLwTzA3v/BhcWl5ZXXt4/rGZm1ru4W2cBKa0mrrrlKBoFUGTVKk4Sp3IEyq4TIdnI/7lzfgUNnsgoY5JEZcZ6qnpCBvdWqH8fdOGXeFG+SFyzWUDX91htsUq6pqx9oi7sfUBxIn/NfnpFOrhwfhpPh7Ec1Enc2q0amN4q6VhYGMpBaI7SjMKSmFIyU1VKtxgZALORDX0PYyEwYwKScfq/ied7q8Z50/GfGJ+3KiFAZxaFKfNIL6+LY3Nv/XaxfUO05KleUFQSani3qF5mT5mBLvKgeS9NALIZ3yb+WyL5yQ5Fm+2oLkSaIP4BfjoxYrjyh6C+S9aB0dRF7//Fo/PbudwlpmO2yX7bOIfWOn7AdrsCaT7I49sEf2O7gP/gZPwfM0OhfMAH9iryoY/QNWqa43</latexit>

PA<latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit>

PA<latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit><latexit sha1_base64="FxgM53F7cIbbY38ngTAqOiThXWQ=">AAACFnicbZDLSgMxGIUz9VbrpaMu3QSL4MYyI4IuW9y4rGBboTMMmTTThiaZIRehDAUfQ3Crr+FO3Lr1LXwEM20XtvWHwOGcE/6fL84YVdrzvp3S2vrG5lZ5u7Kzu7dfdQ8OOyo1EpM2TlkqH2KkCKOCtDXVjDxkkiAeM9KNRzdF3n0kUtFU3OtxRkKOBoImFCNtrcitBjEzJA84zluTqDmJ3JpX96YDV4U/FzUwn1bk/gT9FBtOhMYMKdXzvUyHOZKaYkYmlcAokiE8QgPSs1IgTlSYTw+fwFPr9GGSSvuEhlP3748ccaXGPLZNjvRQLWeF+V/WMzq5DnMqMqOJwLNFiWFQp7CgAPtUEqzZ2AqEJbW3QjxEEmFtWS1sUdqSUragzrmtpqpA5C8DWRWdi7pv9d1lrdF8msEqg2NwAs6AD65AA9yCFmgDDAx4Aa/gzXl23p0P53NWLTlzwEdgYZyvX34eoO4=</latexit>

supPA2PA

EA⇠PACVaRA(✓;P )<latexit sha1_base64="nsmM8TG4WD5634r/7hiQI4r341M=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLoOAmQxBctuJMAlNDU119J1NMPZqq2+JQFPjz/Av+BkHc6dKah2ISLxQczjnFvXx1q6THPP/ay65cvXb9xs7N3Vu379zd6+/fG3vbOQEjYZV1pzX3oKSBEUpUcNo64LpWcFIvjlf5yUdwXlrzAZctTDU/M3ImBcdkVf2a+a6tAqtVB6GshpRJQ5kWoYzVMEbK3vwJU+SlpuXGbrhbOGgC07X9FI7H/H38W4zxgOEckL+m5VMaq/4gP8zXQy+LYisGZDtl1f/BGis6DQaF4t5PirzFaeAOpVAQd1nnoeViwc9gkqThGvw0rFlE+iQ5DZ1Zl55Bunb//RG49n6p69TUHOf+YrYy/5dNOpy9mgZp2g7BiM2iWacoWroCSxvpQKBaJsGFk+lWKubccYEJ/7ktHhN8nwr+mU5V61eIiotALovx88Mi6XcvBkfDzxtYO+QReUwOSEFekiPylpRkRAT5Qr6Tn+RX71u2nz3IHm6qWW8L+D45N9ngN7GEvpE=</latexit><latexit sha1_base64="nsmM8TG4WD5634r/7hiQI4r341M=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLoOAmQxBctuJMAlNDU119J1NMPZqq2+JQFPjz/Av+BkHc6dKah2ISLxQczjnFvXx1q6THPP/ay65cvXb9xs7N3Vu379zd6+/fG3vbOQEjYZV1pzX3oKSBEUpUcNo64LpWcFIvjlf5yUdwXlrzAZctTDU/M3ImBcdkVf2a+a6tAqtVB6GshpRJQ5kWoYzVMEbK3vwJU+SlpuXGbrhbOGgC07X9FI7H/H38W4zxgOEckL+m5VMaq/4gP8zXQy+LYisGZDtl1f/BGis6DQaF4t5PirzFaeAOpVAQd1nnoeViwc9gkqThGvw0rFlE+iQ5DZ1Zl55Bunb//RG49n6p69TUHOf+YrYy/5dNOpy9mgZp2g7BiM2iWacoWroCSxvpQKBaJsGFk+lWKubccYEJ/7ktHhN8nwr+mU5V61eIiotALovx88Mi6XcvBkfDzxtYO+QReUwOSEFekiPylpRkRAT5Qr6Tn+RX71u2nz3IHm6qWW8L+D45N9ngN7GEvpE=</latexit><latexit sha1_base64="nsmM8TG4WD5634r/7hiQI4r341M=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLoOAmQxBctuJMAlNDU119J1NMPZqq2+JQFPjz/Av+BkHc6dKah2ISLxQczjnFvXx1q6THPP/ay65cvXb9xs7N3Vu379zd6+/fG3vbOQEjYZV1pzX3oKSBEUpUcNo64LpWcFIvjlf5yUdwXlrzAZctTDU/M3ImBcdkVf2a+a6tAqtVB6GshpRJQ5kWoYzVMEbK3vwJU+SlpuXGbrhbOGgC07X9FI7H/H38W4zxgOEckL+m5VMaq/4gP8zXQy+LYisGZDtl1f/BGis6DQaF4t5PirzFaeAOpVAQd1nnoeViwc9gkqThGvw0rFlE+iQ5DZ1Zl55Bunb//RG49n6p69TUHOf+YrYy/5dNOpy9mgZp2g7BiM2iWacoWroCSxvpQKBaJsGFk+lWKubccYEJ/7ktHhN8nwr+mU5V61eIiotALovx88Mi6XcvBkfDzxtYO+QReUwOSEFekiPylpRkRAT5Qr6Tn+RX71u2nz3IHm6qWW8L+D45N9ngN7GEvpE=</latexit><latexit sha1_base64="nsmM8TG4WD5634r/7hiQI4r341M=">AAACaXicbZBLaxRBFIVr2leMj0x0I7opHIS4MHSLoOAmQxBctuJMAlNDU119J1NMPZqq2+JQFPjz/Av+BkHc6dKah2ISLxQczjnFvXx1q6THPP/ay65cvXb9xs7N3Vu379zd6+/fG3vbOQEjYZV1pzX3oKSBEUpUcNo64LpWcFIvjlf5yUdwXlrzAZctTDU/M3ImBcdkVf2a+a6tAqtVB6GshpRJQ5kWoYzVMEbK3vwJU+SlpuXGbrhbOGgC07X9FI7H/H38W4zxgOEckL+m5VMaq/4gP8zXQy+LYisGZDtl1f/BGis6DQaF4t5PirzFaeAOpVAQd1nnoeViwc9gkqThGvw0rFlE+iQ5DZ1Zl55Bunb//RG49n6p69TUHOf+YrYy/5dNOpy9mgZp2g7BiM2iWacoWroCSxvpQKBaJsGFk+lWKubccYEJ/7ktHhN8nwr+mU5V61eIiotALovx88Mi6XcvBkfDzxtYO+QReUwOSEFekiPylpRkRAT5Qr6Tn+RX71u2nz3IHm6qWW8L+D45N9ngN7GEvpE=</latexit>

Goal: Tractable family of worst-case performance

maxz

`(✓, z)

Page 24: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Population

Minority Subpopulation

Population

Minority Subpopulation

↵<latexit sha1_base64="z/wN4rOrQs3XID5OHa5ZWjjV+KM=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgi4DblxJBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOPb3O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNe6LssrgBJyCcxCAa9AAd6AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0CjJ+fDQ==</latexit><latexit sha1_base64="z/wN4rOrQs3XID5OHa5ZWjjV+KM=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgi4DblxJBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOPb3O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNe6LssrgBJyCcxCAa9AAd6AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0CjJ+fDQ==</latexit><latexit sha1_base64="z/wN4rOrQs3XID5OHa5ZWjjV+KM=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgi4DblxJBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOPb3O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNe6LssrgBJyCcxCAa9AAd6AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0CjJ+fDQ==</latexit><latexit sha1_base64="z/wN4rOrQs3XID5OHa5ZWjjV+KM=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgi4DblxJBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOPb3O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNe6LssrgBJyCcxCAa9AAd6AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0CjJ+fDQ==</latexit>

↵<latexit sha1_base64="q99/Cv1WEYPJHSsVgpJ7/Aqgi8A=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgq4k4MZlBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOO73O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNW6LssrgBJyCcxCAa9AA96AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0Ch8+e/Q==</latexit><latexit sha1_base64="q99/Cv1WEYPJHSsVgpJ7/Aqgi8A=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgq4k4MZlBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOO73O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNW6LssrgBJyCcxCAa9AA96AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0Ch8+e/Q==</latexit><latexit sha1_base64="q99/Cv1WEYPJHSsVgpJ7/Aqgi8A=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgq4k4MZlBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOO73O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNW6LssrgBJyCcxCAa9AA96AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0Ch8+e/Q==</latexit><latexit sha1_base64="q99/Cv1WEYPJHSsVgpJ7/Aqgi8A=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgq4k4MZlBPOBzBB6ejpJk/7R3SOEIacQ3Oo13IlbT+AtPII9ySxMYsGDoqoe71GxYtRY3//2SmvrG5tb5e3Kzu7e/kH18KhtZKoxaWHJpO7GyBBGBWlZahnpKk0QjxnpxOO73O88EW2oFI92okjE0VDQAcXIOikMNUmyEDE1QtN+tebX/RngKgkKUgMFmv3qT5hInHIiLGbImF7gKxtlSFuKGZlWwtQQhfAYDUnPUYE4MVE2+3kKz5ySwIHUboSFM/XvRoa4MRMeuyRHdmSWvVz8z+uldnATZVSo1BKB54cGKYNWwrwAmFBNsGUTRxDW1P0K8QhphK2raeGKsa4k4wLmgruoNHlFwXIhq6R9WQ8cf7iqNW6LssrgBJyCcxCAa9AA96AJWgADBV7AK3jznr1378P7nEdLXrFzDBbgff0Ch8+e/Q==</latexit>

p ",↵ #<latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit>

More risk-averseLess risk-averse

min✓2⇥

EPobs [`(✓;Z)]<latexit sha1_base64="FDCs5bpTpRObDhmgaUA9nwxtnk8=">AAACT3icbZDLShxBFIarx8R4iTrRZTaFg6ALpVsEA9mMhIDLCTgqTjdDdc0Zp5i6NHVOB4amIU/kuwTcuIjPkF12ITWXhZf8UPDVf87hVP15oRVSHD9EjaU3b5ffrayurb/f2Nxqfti+RFd6CV3ptPPXuUDQykKXFGm4LjwIk2u4ysdfpvWr7+BROXtBkwIyI26tGiopKFj95nlqlO1XKY2ABE+V5enFFGuefg32QPhxUfpCQ9UJV2+4y7Gu616qHeL+fOwzvznI+s1WfBTPxF9DsoAWW6jTb/5OB06WBixJLRB7SVxQVglPSmqo19ISoRByLG6hF9AKA5hVsx/XfC84Az50PhxLfOY+naiEQZyYPHQaQSN8WZua/6v1Shp+yipli5LAyvmiYak5OT6Njw+UB0l6EkBIr8JbuRwJLySFkJ9tQQoRY2jAQxNaHdYhouRlIK/h8vgoCfztpNU++zEPa4V9ZLtsnyXslLXZOeuwLpPsjt2zX+wx+hn9if42Frk2ogXssGdqrP4DPEK2gQ==</latexit><latexit sha1_base64="FDCs5bpTpRObDhmgaUA9nwxtnk8=">AAACT3icbZDLShxBFIarx8R4iTrRZTaFg6ALpVsEA9mMhIDLCTgqTjdDdc0Zp5i6NHVOB4amIU/kuwTcuIjPkF12ITWXhZf8UPDVf87hVP15oRVSHD9EjaU3b5ffrayurb/f2Nxqfti+RFd6CV3ptPPXuUDQykKXFGm4LjwIk2u4ysdfpvWr7+BROXtBkwIyI26tGiopKFj95nlqlO1XKY2ABE+V5enFFGuefg32QPhxUfpCQ9UJV2+4y7Gu616qHeL+fOwzvznI+s1WfBTPxF9DsoAWW6jTb/5OB06WBixJLRB7SVxQVglPSmqo19ISoRByLG6hF9AKA5hVsx/XfC84Az50PhxLfOY+naiEQZyYPHQaQSN8WZua/6v1Shp+yipli5LAyvmiYak5OT6Njw+UB0l6EkBIr8JbuRwJLySFkJ9tQQoRY2jAQxNaHdYhouRlIK/h8vgoCfztpNU++zEPa4V9ZLtsnyXslLXZOeuwLpPsjt2zX+wx+hn9if42Frk2ogXssGdqrP4DPEK2gQ==</latexit><latexit sha1_base64="FDCs5bpTpRObDhmgaUA9nwxtnk8=">AAACT3icbZDLShxBFIarx8R4iTrRZTaFg6ALpVsEA9mMhIDLCTgqTjdDdc0Zp5i6NHVOB4amIU/kuwTcuIjPkF12ITWXhZf8UPDVf87hVP15oRVSHD9EjaU3b5ffrayurb/f2Nxqfti+RFd6CV3ptPPXuUDQykKXFGm4LjwIk2u4ysdfpvWr7+BROXtBkwIyI26tGiopKFj95nlqlO1XKY2ABE+V5enFFGuefg32QPhxUfpCQ9UJV2+4y7Gu616qHeL+fOwzvznI+s1WfBTPxF9DsoAWW6jTb/5OB06WBixJLRB7SVxQVglPSmqo19ISoRByLG6hF9AKA5hVsx/XfC84Az50PhxLfOY+naiEQZyYPHQaQSN8WZua/6v1Shp+yipli5LAyvmiYak5OT6Njw+UB0l6EkBIr8JbuRwJLySFkJ9tQQoRY2jAQxNaHdYhouRlIK/h8vgoCfztpNU++zEPa4V9ZLtsnyXslLXZOeuwLpPsjt2zX+wx+hn9if42Frk2ogXssGdqrP4DPEK2gQ==</latexit><latexit sha1_base64="FDCs5bpTpRObDhmgaUA9nwxtnk8=">AAACT3icbZDLShxBFIarx8R4iTrRZTaFg6ALpVsEA9mMhIDLCTgqTjdDdc0Zp5i6NHVOB4amIU/kuwTcuIjPkF12ITWXhZf8UPDVf87hVP15oRVSHD9EjaU3b5ffrayurb/f2Nxqfti+RFd6CV3ptPPXuUDQykKXFGm4LjwIk2u4ysdfpvWr7+BROXtBkwIyI26tGiopKFj95nlqlO1XKY2ABE+V5enFFGuefg32QPhxUfpCQ9UJV2+4y7Gu616qHeL+fOwzvznI+s1WfBTPxF9DsoAWW6jTb/5OB06WBixJLRB7SVxQVglPSmqo19ISoRByLG6hF9AKA5hVsx/XfC84Az50PhxLfOY+naiEQZyYPHQaQSN8WZua/6v1Shp+yipli5LAyvmiYak5OT6Njw+UB0l6EkBIr8JbuRwJLySFkJ9tQQoRY2jAQxNaHdYhouRlIK/h8vgoCfztpNU++zEPa4V9ZLtsnyXslLXZOeuwLpPsjt2zX+wx+hn9if42Frk2ogXssGdqrP4DPEK2gQ==</latexit>

Average-case Worst-off individual min✓2⇥

supZ

`(✓;Z)<latexit sha1_base64="MUTJttZFQEcL9C4RYkUJs9pFDHI=">AAACRnicbZBLaxRBFIVvj68YX6MusykchLgwdAchgSwSceMygUwSMjU01TU1mWLqRd3bwtC0+Gf8L4JbXfgnJJvg1prHwiReKPg45xS36lTBaKQ8/5V17ty9d//B2sP1R4+fPH3Wff7iBH0dpepLb3w8qwQqo53qkyajzkJUwlZGnVbTD3P/9JOKqL07pllQQysunB5rKShJZXefW+3KhtNEkWBcO8aP59gyPhJxGuoYjGo41qFsztv2MzcecXMZ32Pnb8puL9/KF8NuQ7GCHqzmsOxe8pGXtVWOpBGIgyIPNGxEJC2Natd5jSoIORUXapDQCatw2Cw+2rLXSRmxsY/pOGIL9d8bjbCIM1ulpBU0wZveXPyfN6hpvDtstAs1KSeXi8a1YeTZvDU20lFJMrMEQkad3srkREQhKXV7bQtSahZTAN/aFPXYpoqKm4XchpPtrSLx0bvewfsvy7LWYANewSYUsAMH8BEOoQ8SvsJ3+AE/s2/Z7+wq+7OMdrJVwS/h2nTgLydRtBM=</latexit><latexit sha1_base64="MUTJttZFQEcL9C4RYkUJs9pFDHI=">AAACRnicbZBLaxRBFIVvj68YX6MusykchLgwdAchgSwSceMygUwSMjU01TU1mWLqRd3bwtC0+Gf8L4JbXfgnJJvg1prHwiReKPg45xS36lTBaKQ8/5V17ty9d//B2sP1R4+fPH3Wff7iBH0dpepLb3w8qwQqo53qkyajzkJUwlZGnVbTD3P/9JOKqL07pllQQysunB5rKShJZXefW+3KhtNEkWBcO8aP59gyPhJxGuoYjGo41qFsztv2MzcecXMZ32Pnb8puL9/KF8NuQ7GCHqzmsOxe8pGXtVWOpBGIgyIPNGxEJC2Natd5jSoIORUXapDQCatw2Cw+2rLXSRmxsY/pOGIL9d8bjbCIM1ulpBU0wZveXPyfN6hpvDtstAs1KSeXi8a1YeTZvDU20lFJMrMEQkad3srkREQhKXV7bQtSahZTAN/aFPXYpoqKm4XchpPtrSLx0bvewfsvy7LWYANewSYUsAMH8BEOoQ8SvsJ3+AE/s2/Z7+wq+7OMdrJVwS/h2nTgLydRtBM=</latexit><latexit sha1_base64="MUTJttZFQEcL9C4RYkUJs9pFDHI=">AAACRnicbZBLaxRBFIVvj68YX6MusykchLgwdAchgSwSceMygUwSMjU01TU1mWLqRd3bwtC0+Gf8L4JbXfgnJJvg1prHwiReKPg45xS36lTBaKQ8/5V17ty9d//B2sP1R4+fPH3Wff7iBH0dpepLb3w8qwQqo53qkyajzkJUwlZGnVbTD3P/9JOKqL07pllQQysunB5rKShJZXefW+3KhtNEkWBcO8aP59gyPhJxGuoYjGo41qFsztv2MzcecXMZ32Pnb8puL9/KF8NuQ7GCHqzmsOxe8pGXtVWOpBGIgyIPNGxEJC2Natd5jSoIORUXapDQCatw2Cw+2rLXSRmxsY/pOGIL9d8bjbCIM1ulpBU0wZveXPyfN6hpvDtstAs1KSeXi8a1YeTZvDU20lFJMrMEQkad3srkREQhKXV7bQtSahZTAN/aFPXYpoqKm4XchpPtrSLx0bvewfsvy7LWYANewSYUsAMH8BEOoQ8SvsJ3+AE/s2/Z7+wq+7OMdrJVwS/h2nTgLydRtBM=</latexit><latexit sha1_base64="MUTJttZFQEcL9C4RYkUJs9pFDHI=">AAACRnicbZBLaxRBFIVvj68YX6MusykchLgwdAchgSwSceMygUwSMjU01TU1mWLqRd3bwtC0+Gf8L4JbXfgnJJvg1prHwiReKPg45xS36lTBaKQ8/5V17ty9d//B2sP1R4+fPH3Wff7iBH0dpepLb3w8qwQqo53qkyajzkJUwlZGnVbTD3P/9JOKqL07pllQQysunB5rKShJZXefW+3KhtNEkWBcO8aP59gyPhJxGuoYjGo41qFsztv2MzcecXMZ32Pnb8puL9/KF8NuQ7GCHqzmsOxe8pGXtVWOpBGIgyIPNGxEJC2Natd5jSoIORUXapDQCatw2Cw+2rLXSRmxsY/pOGIL9d8bjbCIM1ulpBU0wZveXPyfN6hpvDtstAs1KSeXi8a1YeTZvDU20lFJMrMEQkad3srkREQhKXV7bQtSahZTAN/aFPXYpoqKm4XchpPtrSLx0bvewfsvy7LWYANewSYUsAMH8BEOoQ8SvsJ3+AE/s2/Z7+wq+7OMdrJVwS/h2nTgLydRtBM=</latexit>

interpolate via p ",↵ #<latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit><latexit sha1_base64="GNAefZnKrcWV5W5J+FRTJKwpxEQ=">AAACNnicbVDLSgMxFM34flt16SZYBBc+ZkTQlQhuXFawKnRKuZO5bUMzSUgyljL0M/wXwa1+hBt34rafYPpY+DoQOJxzLvfmJFpw68LwLZianpmdm19YXFpeWV1bL21s3lqVG4ZVpoQy9wlYFFxi1XEn8F4bhCwReJd0Lof+3QMay5W8cT2N9Qxakjc5A+elRukoTsF0WgZRFprGuQZjVLe/T2ODaRGD0G2gcaq6cmw0SuXwMByB/iXRhJTJBJVGaeCnWZ6hdEyAtbUo1K5egHGcCewvxblFDawDLax5KiFDWy9GH+vTXa+ktKmMf9LRkfp9ooDM2l6W+GQGrm1/e0PxP6+Wu+ZZveBS5w4lGy9q5oI6RYct0ZQbZE70PAFmuL+VsjYYYM53+WOLdb5J6wP2IPNRZYcVRb8L+Utujw8jz69Pyhfnk7IWyDbZIXskIqfkglyRCqkSRh7JM3khr8FT8B58BJ/j6FQwmdkiPxAMvgADqa46</latexit>

Approach: higher-order CVaR

minimize✓2⇥

⇢Rp,↵(✓;Pobs) = inf

⇢1

�EPobs(`(✓;Z)� ⌘)p+

� 1p + ⌘

��

<latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit>

minimize✓2⇥

EPobs [`(✓;Z)]<latexit sha1_base64="8b6zGY4NkagqabLKtQBbMVviFm8=">AAACVnicbZBNaxRBEIZ7J8bE9SNjcvTSuAjxYJgRIYIggSB4c4VsEtwehp7e2myT/hi6aoR1mPwr/4viNd79BWLvZg/5sKDh4a23qOq3qo1GyrKfvWTt3vr9jc0H/YePHj/ZSp9uH6NvgoKR8saH00oiGO1gRJoMnNYBpK0MnFTnh4v+yVcIqL07onkNhZVnTk+1khSlMv0krHba6m9QtqIyDbSCZkCSC+24OFpg1/UvxIeyHUZHsNxX2HVjYTzi7vWJ7h3/8rIo00G2ly2L34V8BQO2qmGZ/hETrxoLjpSRiOM8q6loZSCtDHR90SDUUp3LMxhHdNICFu3y4x1/EZUJn/oQnyO+VK9PtNIizm0VnVbSDG/3FuL/euOGpm+LVru6IXDqatG0MZw8X6TIJzqAIjOPIFXQ8VauZjJIRTHrG1uQYtIYDfjKRqvHLkaU3w7kLhy/3ssjf34zOHi/CmuTPWPP2S7L2T47YB/ZkI2YYt/ZL3bJfvd+9P4m68nGlTXprWZ22I1K0n+k6rg1</latexit><latexit sha1_base64="8b6zGY4NkagqabLKtQBbMVviFm8=">AAACVnicbZBNaxRBEIZ7J8bE9SNjcvTSuAjxYJgRIYIggSB4c4VsEtwehp7e2myT/hi6aoR1mPwr/4viNd79BWLvZg/5sKDh4a23qOq3qo1GyrKfvWTt3vr9jc0H/YePHj/ZSp9uH6NvgoKR8saH00oiGO1gRJoMnNYBpK0MnFTnh4v+yVcIqL07onkNhZVnTk+1khSlMv0krHba6m9QtqIyDbSCZkCSC+24OFpg1/UvxIeyHUZHsNxX2HVjYTzi7vWJ7h3/8rIo00G2ly2L34V8BQO2qmGZ/hETrxoLjpSRiOM8q6loZSCtDHR90SDUUp3LMxhHdNICFu3y4x1/EZUJn/oQnyO+VK9PtNIizm0VnVbSDG/3FuL/euOGpm+LVru6IXDqatG0MZw8X6TIJzqAIjOPIFXQ8VauZjJIRTHrG1uQYtIYDfjKRqvHLkaU3w7kLhy/3ssjf34zOHi/CmuTPWPP2S7L2T47YB/ZkI2YYt/ZL3bJfvd+9P4m68nGlTXprWZ22I1K0n+k6rg1</latexit><latexit sha1_base64="8b6zGY4NkagqabLKtQBbMVviFm8=">AAACVnicbZBNaxRBEIZ7J8bE9SNjcvTSuAjxYJgRIYIggSB4c4VsEtwehp7e2myT/hi6aoR1mPwr/4viNd79BWLvZg/5sKDh4a23qOq3qo1GyrKfvWTt3vr9jc0H/YePHj/ZSp9uH6NvgoKR8saH00oiGO1gRJoMnNYBpK0MnFTnh4v+yVcIqL07onkNhZVnTk+1khSlMv0krHba6m9QtqIyDbSCZkCSC+24OFpg1/UvxIeyHUZHsNxX2HVjYTzi7vWJ7h3/8rIo00G2ly2L34V8BQO2qmGZ/hETrxoLjpSRiOM8q6loZSCtDHR90SDUUp3LMxhHdNICFu3y4x1/EZUJn/oQnyO+VK9PtNIizm0VnVbSDG/3FuL/euOGpm+LVru6IXDqatG0MZw8X6TIJzqAIjOPIFXQ8VauZjJIRTHrG1uQYtIYDfjKRqvHLkaU3w7kLhy/3ssjf34zOHi/CmuTPWPP2S7L2T47YB/ZkI2YYt/ZL3bJfvd+9P4m68nGlTXprWZ22I1K0n+k6rg1</latexit><latexit sha1_base64="8b6zGY4NkagqabLKtQBbMVviFm8=">AAACVnicbZBNaxRBEIZ7J8bE9SNjcvTSuAjxYJgRIYIggSB4c4VsEtwehp7e2myT/hi6aoR1mPwr/4viNd79BWLvZg/5sKDh4a23qOq3qo1GyrKfvWTt3vr9jc0H/YePHj/ZSp9uH6NvgoKR8saH00oiGO1gRJoMnNYBpK0MnFTnh4v+yVcIqL07onkNhZVnTk+1khSlMv0krHba6m9QtqIyDbSCZkCSC+24OFpg1/UvxIeyHUZHsNxX2HVjYTzi7vWJ7h3/8rIo00G2ly2L34V8BQO2qmGZ/hETrxoLjpSRiOM8q6loZSCtDHR90SDUUp3LMxhHdNICFu3y4x1/EZUJn/oQnyO+VK9PtNIizm0VnVbSDG/3FuL/euOGpm+LVru6IXDqatG0MZw8X6TIJzqAIjOPIFXQ8VauZjJIRTHrG1uQYtIYDfjKRqvHLkaU3w7kLhy/3ssjf34zOHi/CmuTPWPP2S7L2T47YB/ZkI2YYt/ZL3bJfvd+9P4m68nGlTXprWZ22I1K0n+k6rg1</latexit>

Page 25: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Convexity

convex loss ✓ 7! `(✓;Z)

<latexit sha1_base64="zwb7vplPt+WNX69qeINz62Gzp7o=">AAACNHicbZDNSgMxFIUz/v9bdekmWARdKDMiKLhR3LhUsCp2SsmktzaYTIbcO0IZCr6D7yK41acQ3Ilb8RFMfxZWvRA4nHPCTb4k0wopDF+DkdGx8YnJqemZ2bn5hcXS0vIF2txJqEirrbtKBIJWKVRIkYarzIEwiYbL5Pa4m1/egUNl03NqZ1Az4iZVTSUFeate2ooTnUMRUwtIdHhsRIZkeawt4sZQdsCvN+ulcrgd9ob/FdFAlNlgTuulr7hhZW4gJakFYjUKM6oVwpGSGjozcY6QCXkrbqDqZSoMYK3ofavD173T4E3r/EmJ99yfNwphENsm8U0jqIW/s675X1bNqblfK1Sa5QSp7C9q5pr7j3cZ8YZyIEm3vRDSKf9WLlvCCUme5NAWJM8RfQG3jK9a7HhE0W8gf8XFznbk9dlu+fDovg9riq2yNbbBIrbHDtkJO2UVJtkDe2LP7CV4DN6C9+CjXx0JBoBX2NAEn99lAa05</latexit><latexit sha1_base64="zwb7vplPt+WNX69qeINz62Gzp7o=">AAACNHicbZDNSgMxFIUz/v9bdekmWARdKDMiKLhR3LhUsCp2SsmktzaYTIbcO0IZCr6D7yK41acQ3Ilb8RFMfxZWvRA4nHPCTb4k0wopDF+DkdGx8YnJqemZ2bn5hcXS0vIF2txJqEirrbtKBIJWKVRIkYarzIEwiYbL5Pa4m1/egUNl03NqZ1Az4iZVTSUFeate2ooTnUMRUwtIdHhsRIZkeawt4sZQdsCvN+ulcrgd9ob/FdFAlNlgTuulr7hhZW4gJakFYjUKM6oVwpGSGjozcY6QCXkrbqDqZSoMYK3ofavD173T4E3r/EmJ99yfNwphENsm8U0jqIW/s675X1bNqblfK1Sa5QSp7C9q5pr7j3cZ8YZyIEm3vRDSKf9WLlvCCUme5NAWJM8RfQG3jK9a7HhE0W8gf8XFznbk9dlu+fDovg9riq2yNbbBIrbHDtkJO2UVJtkDe2LP7CV4DN6C9+CjXx0JBoBX2NAEn99lAa05</latexit><latexit sha1_base64="zwb7vplPt+WNX69qeINz62Gzp7o=">AAACNHicbZDNSgMxFIUz/v9bdekmWARdKDMiKLhR3LhUsCp2SsmktzaYTIbcO0IZCr6D7yK41acQ3Ilb8RFMfxZWvRA4nHPCTb4k0wopDF+DkdGx8YnJqemZ2bn5hcXS0vIF2txJqEirrbtKBIJWKVRIkYarzIEwiYbL5Pa4m1/egUNl03NqZ1Az4iZVTSUFeate2ooTnUMRUwtIdHhsRIZkeawt4sZQdsCvN+ulcrgd9ob/FdFAlNlgTuulr7hhZW4gJakFYjUKM6oVwpGSGjozcY6QCXkrbqDqZSoMYK3ofavD173T4E3r/EmJ99yfNwphENsm8U0jqIW/s675X1bNqblfK1Sa5QSp7C9q5pr7j3cZ8YZyIEm3vRDSKf9WLlvCCUme5NAWJM8RfQG3jK9a7HhE0W8gf8XFznbk9dlu+fDovg9riq2yNbbBIrbHDtkJO2UVJtkDe2LP7CV4DN6C9+CjXx0JBoBX2NAEn99lAa05</latexit><latexit sha1_base64="zwb7vplPt+WNX69qeINz62Gzp7o=">AAACNHicbZDNSgMxFIUz/v9bdekmWARdKDMiKLhR3LhUsCp2SsmktzaYTIbcO0IZCr6D7yK41acQ3Ilb8RFMfxZWvRA4nHPCTb4k0wopDF+DkdGx8YnJqemZ2bn5hcXS0vIF2txJqEirrbtKBIJWKVRIkYarzIEwiYbL5Pa4m1/egUNl03NqZ1Az4iZVTSUFeate2ooTnUMRUwtIdHhsRIZkeawt4sZQdsCvN+ulcrgd9ob/FdFAlNlgTuulr7hhZW4gJakFYjUKM6oVwpGSGjozcY6QCXkrbqDqZSoMYK3ofavD173T4E3r/EmJ99yfNwphENsm8U0jqIW/s675X1bNqblfK1Sa5QSp7C9q5pr7j3cZ8YZyIEm3vRDSKf9WLlvCCUme5NAWJM8RfQG3jK9a7HhE0W8gf8XFznbk9dlu+fDovg9riq2yNbbBIrbHDtkJO2UVJtkDe2LP7CV4DN6C9+CjXx0JBoBX2NAEn99lAa05</latexit>

convex worst-case risk ✓ 7! Rp,↵(✓; bPobs,n)

<latexit sha1_base64="8vHxhTrJRXQQKHeWgYlY7svLXL4=">AAACVnicbZDPahRBEMZ7R2Ni/Dfq0UvjIkSIYSYICoIEvHhzBTcJbC9DTW9vptnunqarRlmaeSzfRfGqd59A7N3swU0saPj46iuq+ld7o5GK4vsgu3Fz59bu3u39O3fv3X+QP3x0im0XpBrL1rThvAZURjs1Jk1GnfugwNZGndWLd6v+2WcVULfuEy29mlq4cHquJVCyqvyDqE2noqBGEfRcWPBILRdB46KK/pALML6B/mAr94aLLw1QHPVVjCJY3tbYH3LXP6/yYXFUrItfF+VGDNmmRlX+W8xa2VnlSBpAnJSFp2mEQFoa1e+LDpUHuYALNUnSgVU4jeuP9/xZcmZ83ob0HPG1++9EBIu4tHVKWqAGr/ZW5v96k47mr6dRO9+RcvJy0bwzPKFZUeQzHZQks0wCZNDpVi4bCCApsd7agpRIYwrgC5uiLfYJUXkVyHVxenxUJv3x5fDk7QbWHnvCnrIDVrJX7IS9ZyM2ZpJ9ZT/YT/Zr8G3wJ9vJdi+j2WAz85htVZb/BXmQuCA=</latexit><latexit sha1_base64="8vHxhTrJRXQQKHeWgYlY7svLXL4=">AAACVnicbZDPahRBEMZ7R2Ni/Dfq0UvjIkSIYSYICoIEvHhzBTcJbC9DTW9vptnunqarRlmaeSzfRfGqd59A7N3swU0saPj46iuq+ld7o5GK4vsgu3Fz59bu3u39O3fv3X+QP3x0im0XpBrL1rThvAZURjs1Jk1GnfugwNZGndWLd6v+2WcVULfuEy29mlq4cHquJVCyqvyDqE2noqBGEfRcWPBILRdB46KK/pALML6B/mAr94aLLw1QHPVVjCJY3tbYH3LXP6/yYXFUrItfF+VGDNmmRlX+W8xa2VnlSBpAnJSFp2mEQFoa1e+LDpUHuYALNUnSgVU4jeuP9/xZcmZ83ob0HPG1++9EBIu4tHVKWqAGr/ZW5v96k47mr6dRO9+RcvJy0bwzPKFZUeQzHZQks0wCZNDpVi4bCCApsd7agpRIYwrgC5uiLfYJUXkVyHVxenxUJv3x5fDk7QbWHnvCnrIDVrJX7IS9ZyM2ZpJ9ZT/YT/Zr8G3wJ9vJdi+j2WAz85htVZb/BXmQuCA=</latexit><latexit sha1_base64="8vHxhTrJRXQQKHeWgYlY7svLXL4=">AAACVnicbZDPahRBEMZ7R2Ni/Dfq0UvjIkSIYSYICoIEvHhzBTcJbC9DTW9vptnunqarRlmaeSzfRfGqd59A7N3swU0saPj46iuq+ld7o5GK4vsgu3Fz59bu3u39O3fv3X+QP3x0im0XpBrL1rThvAZURjs1Jk1GnfugwNZGndWLd6v+2WcVULfuEy29mlq4cHquJVCyqvyDqE2noqBGEfRcWPBILRdB46KK/pALML6B/mAr94aLLw1QHPVVjCJY3tbYH3LXP6/yYXFUrItfF+VGDNmmRlX+W8xa2VnlSBpAnJSFp2mEQFoa1e+LDpUHuYALNUnSgVU4jeuP9/xZcmZ83ob0HPG1++9EBIu4tHVKWqAGr/ZW5v96k47mr6dRO9+RcvJy0bwzPKFZUeQzHZQks0wCZNDpVi4bCCApsd7agpRIYwrgC5uiLfYJUXkVyHVxenxUJv3x5fDk7QbWHnvCnrIDVrJX7IS9ZyM2ZpJ9ZT/YT/Zr8G3wJ9vJdi+j2WAz85htVZb/BXmQuCA=</latexit><latexit sha1_base64="8vHxhTrJRXQQKHeWgYlY7svLXL4=">AAACVnicbZDPahRBEMZ7R2Ni/Dfq0UvjIkSIYSYICoIEvHhzBTcJbC9DTW9vptnunqarRlmaeSzfRfGqd59A7N3swU0saPj46iuq+ld7o5GK4vsgu3Fz59bu3u39O3fv3X+QP3x0im0XpBrL1rThvAZURjs1Jk1GnfugwNZGndWLd6v+2WcVULfuEy29mlq4cHquJVCyqvyDqE2noqBGEfRcWPBILRdB46KK/pALML6B/mAr94aLLw1QHPVVjCJY3tbYH3LXP6/yYXFUrItfF+VGDNmmRlX+W8xa2VnlSBpAnJSFp2mEQFoa1e+LDpUHuYALNUnSgVU4jeuP9/xZcmZ83ob0HPG1++9EBIu4tHVKWqAGr/ZW5v96k47mr6dRO9+RcvJy0bwzPKFZUeQzHZQks0wCZNDpVi4bCCApsd7agpRIYwrgC5uiLfYJUXkVyHVxenxUJv3x5fDk7QbWHnvCnrIDVrJX7IS9ZyM2ZpJ9ZT/YT/Zr8G3wJ9vJdi+j2WAz85htVZb/BXmQuCA=</latexit>

`(✓;X,Y ) =1

2(Y � ✓>X)2

<latexit sha1_base64="vHSY/Pqtw22E0zRWiot+kjTV+/M=">AAACH3icbVDLSgNBEJyN7/ha9ehlMAgRNOwGUUEUwYtHBaMJ2RhmJ73JkNkHM71CWPInXvwVLx4UEW/+jZPHQRMLBoqqanq6/EQKjY7zbeVmZufmFxaX8ssrq2vr9sbmnY5TxaHCYxmrqs80SBFBBQVKqCYKWOhLuPe7lwP//hGUFnF0i70EGiFrRyIQnKGRmvaRB1IWPewAslNa3ae1PXpGvUAxnrn9rNynxRo9oKPAg4dxQqt7D+WmXXBKzhB0mrhjUiBjXDftL68V8zSECLlkWtddJ8FGxhQKLqGf91INCeNd1oa6oRELQTey4X19umuUFg1iZV6EdKj+nshYqHUv9E0yZNjRk95A/M+rpxicNDIRJSlCxEeLglRSjOmgLNoSCjjKniGMK2H+SnmHmW7QVJo3JbiTJ0+Tu3LJNfzmsHBxPq5jkWyTHVIkLjkmF+SKXJMK4eSJvJA38m49W6/Wh/U5iuas8cwW+QPr+wdCR5/9</latexit><latexit sha1_base64="vHSY/Pqtw22E0zRWiot+kjTV+/M=">AAACH3icbVDLSgNBEJyN7/ha9ehlMAgRNOwGUUEUwYtHBaMJ2RhmJ73JkNkHM71CWPInXvwVLx4UEW/+jZPHQRMLBoqqanq6/EQKjY7zbeVmZufmFxaX8ssrq2vr9sbmnY5TxaHCYxmrqs80SBFBBQVKqCYKWOhLuPe7lwP//hGUFnF0i70EGiFrRyIQnKGRmvaRB1IWPewAslNa3ae1PXpGvUAxnrn9rNynxRo9oKPAg4dxQqt7D+WmXXBKzhB0mrhjUiBjXDftL68V8zSECLlkWtddJ8FGxhQKLqGf91INCeNd1oa6oRELQTey4X19umuUFg1iZV6EdKj+nshYqHUv9E0yZNjRk95A/M+rpxicNDIRJSlCxEeLglRSjOmgLNoSCjjKniGMK2H+SnmHmW7QVJo3JbiTJ0+Tu3LJNfzmsHBxPq5jkWyTHVIkLjkmF+SKXJMK4eSJvJA38m49W6/Wh/U5iuas8cwW+QPr+wdCR5/9</latexit><latexit sha1_base64="vHSY/Pqtw22E0zRWiot+kjTV+/M=">AAACH3icbVDLSgNBEJyN7/ha9ehlMAgRNOwGUUEUwYtHBaMJ2RhmJ73JkNkHM71CWPInXvwVLx4UEW/+jZPHQRMLBoqqanq6/EQKjY7zbeVmZufmFxaX8ssrq2vr9sbmnY5TxaHCYxmrqs80SBFBBQVKqCYKWOhLuPe7lwP//hGUFnF0i70EGiFrRyIQnKGRmvaRB1IWPewAslNa3ae1PXpGvUAxnrn9rNynxRo9oKPAg4dxQqt7D+WmXXBKzhB0mrhjUiBjXDftL68V8zSECLlkWtddJ8FGxhQKLqGf91INCeNd1oa6oRELQTey4X19umuUFg1iZV6EdKj+nshYqHUv9E0yZNjRk95A/M+rpxicNDIRJSlCxEeLglRSjOmgLNoSCjjKniGMK2H+SnmHmW7QVJo3JbiTJ0+Tu3LJNfzmsHBxPq5jkWyTHVIkLjkmF+SKXJMK4eSJvJA38m49W6/Wh/U5iuas8cwW+QPr+wdCR5/9</latexit><latexit sha1_base64="vHSY/Pqtw22E0zRWiot+kjTV+/M=">AAACH3icbVDLSgNBEJyN7/ha9ehlMAgRNOwGUUEUwYtHBaMJ2RhmJ73JkNkHM71CWPInXvwVLx4UEW/+jZPHQRMLBoqqanq6/EQKjY7zbeVmZufmFxaX8ssrq2vr9sbmnY5TxaHCYxmrqs80SBFBBQVKqCYKWOhLuPe7lwP//hGUFnF0i70EGiFrRyIQnKGRmvaRB1IWPewAslNa3ae1PXpGvUAxnrn9rNynxRo9oKPAg4dxQqt7D+WmXXBKzhB0mrhjUiBjXDftL68V8zSECLlkWtddJ8FGxhQKLqGf91INCeNd1oa6oRELQTey4X19umuUFg1iZV6EdKj+nshYqHUv9E0yZNjRk95A/M+rpxicNDIRJSlCxEeLglRSjOmgLNoSCjjKniGMK2H+SnmHmW7QVJo3JbiTJ0+Tu3LJNfzmsHBxPq5jkWyTHVIkLjkmF+SKXJMK4eSJvJA38m49W6/Wh/U5iuas8cwW+QPr+wdCR5/9</latexit>

Example:

Conic form representable

Efficient first-order methods[Krokhmal ’07]

[N. & Duchi ’17]

minimize✓2⇥

⇢Rp,↵(✓;Pobs) = inf

⇢1

�EPobs(`(✓;Z)� ⌘)p+

� 1p + ⌘

��

<latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit><latexit sha1_base64="FKCgUe+jln7fr8y2eDwaWi/Qs1o=">AAADOXicbVJdixMxFM2MX+v61dVHX4JFaFl36YiwCyKsiOBjhe3uYlOHTHqnDU0yQ5IRagj4M/wvgq/6F3z0TXwS/ANm2rI77Xph4HDuyTmZm5uVghvb6/2I4itXr12/sXVz+9btO3fvtXbun5ii0gwGrBCFPsuoAcEVDCy3As5KDVRmAk6z2au6f/oBtOGFOrbzEkaSThTPOaM2UOlOlBDJFZf8I6SOZKICR+wULMWEK0yOa+g9JgJySxwmmptZEI6pnk00gHKlfxJYGDtCRTmttZ2mjX+O+0GvJS4y47v4Re2bn0dtutc+uabMJf7CcNHukNepa1j5DhGFMRth77p4Dze8u+nu+/Xb+vALk6ntrtHnmaHv8W7TAS/15AKkrXZvv7cofBkkK9BGq+qnrT9kXLBKgrJMUGOGSa+0I0e15UyA3yaVgZKyGZ3AMEBFJZiRW7ytx48DM8Z5ocOnLF6wzROOSmPmMgtKSe3UbPZq8n+9YWXzw5HjqqwsKLYMyiuBbYHrRcFjroFZMQ+AMs3DXTGb0jAnG9ZpLcXYsEwmCMyeDNLC1CNKNgdyGZw83U8CfvusffTy03JYW+gheoQ6KEEH6Ai9QX00QCz6HH2NvkXf4y/xz/hX/HspjaPVgB+gtYr//gMIrRGT</latexit>

minimize✓2⇥

8<

:Rp,↵(✓; bPobs,n) = inf⌘

8<

:1

1

n

nX

i=1

(`(✓;Zi)� ⌘)p+

! 1p

+ ⌘

9=

;

9=

;<latexit sha1_base64="w2TBmwlAQZj/vRWi63pfItSWfq0=">AAADT3icbVJdixMxFM20fuyuH9vVR1+CRWhZt3REUJCFFR/0scJ2d7HpDpn0ThuaZIYko9QQ8Bf5XwRffNDf4JtvYqYt3bbrhYHDOSf3JHduWghubLf7I6rVb9y8dXtnd+/O3Xv39xsHD85MXmoGfZaLXF+k1IDgCvqWWwEXhQYqUwHn6fRNpZ9/BG14rk7trIChpGPFM86oDVRyEL0lkisu+WdIHElFCY7YCViKCVeYnFbQe0wEZJY4TDQ302AcUT0dawDlCv80sDByhIpiUnlb6238K0w+Tah1PZ84R7TEeWrCEeXb+LjKyFax20lVz0xT5mJ/1Xwut1a88sSUMnH8OPaXKkSL3JitC3xIeBsf4bWUdnJ4ufkGHx42ntj2Br1KCbrHh+sd8MJPrkDSaHY73Xnh6yBegiZaVi9p/CajnJUSlGWCGjOIu4UdOqotZwL8HikNFJRN6RgGASoqwQzd/I97/CQwI5zlOnzK4jm7fsJRacxMpsEpqZ2Yba0i/6cNSpu9HDquitKCYougrBTY5rhaHzziGpgVswAo0zzcFbMJDXOyYck2UowNK2aCwRzJYM1NNaJ4eyDXwdmzThx34vfPmyevvyyGtYMeoceohWL0Ap2gd6iH+ohFX6Pv0c/oV+1b7U/tb30511q0BA/RRtV3/wHT1xkI</latexit><latexit sha1_base64="w2TBmwlAQZj/vRWi63pfItSWfq0=">AAADT3icbVJdixMxFM20fuyuH9vVR1+CRWhZt3REUJCFFR/0scJ2d7HpDpn0ThuaZIYko9QQ8Bf5XwRffNDf4JtvYqYt3bbrhYHDOSf3JHduWghubLf7I6rVb9y8dXtnd+/O3Xv39xsHD85MXmoGfZaLXF+k1IDgCvqWWwEXhQYqUwHn6fRNpZ9/BG14rk7trIChpGPFM86oDVRyEL0lkisu+WdIHElFCY7YCViKCVeYnFbQe0wEZJY4TDQ302AcUT0dawDlCv80sDByhIpiUnlb6238K0w+Tah1PZ84R7TEeWrCEeXb+LjKyFax20lVz0xT5mJ/1Xwut1a88sSUMnH8OPaXKkSL3JitC3xIeBsf4bWUdnJ4ufkGHx42ntj2Br1KCbrHh+sd8MJPrkDSaHY73Xnh6yBegiZaVi9p/CajnJUSlGWCGjOIu4UdOqotZwL8HikNFJRN6RgGASoqwQzd/I97/CQwI5zlOnzK4jm7fsJRacxMpsEpqZ2Yba0i/6cNSpu9HDquitKCYougrBTY5rhaHzziGpgVswAo0zzcFbMJDXOyYck2UowNK2aCwRzJYM1NNaJ4eyDXwdmzThx34vfPmyevvyyGtYMeoceohWL0Ap2gd6iH+ohFX6Pv0c/oV+1b7U/tb30511q0BA/RRtV3/wHT1xkI</latexit><latexit sha1_base64="w2TBmwlAQZj/vRWi63pfItSWfq0=">AAADT3icbVJdixMxFM20fuyuH9vVR1+CRWhZt3REUJCFFR/0scJ2d7HpDpn0ThuaZIYko9QQ8Bf5XwRffNDf4JtvYqYt3bbrhYHDOSf3JHduWghubLf7I6rVb9y8dXtnd+/O3Xv39xsHD85MXmoGfZaLXF+k1IDgCvqWWwEXhQYqUwHn6fRNpZ9/BG14rk7trIChpGPFM86oDVRyEL0lkisu+WdIHElFCY7YCViKCVeYnFbQe0wEZJY4TDQ302AcUT0dawDlCv80sDByhIpiUnlb6238K0w+Tah1PZ84R7TEeWrCEeXb+LjKyFax20lVz0xT5mJ/1Xwut1a88sSUMnH8OPaXKkSL3JitC3xIeBsf4bWUdnJ4ufkGHx42ntj2Br1KCbrHh+sd8MJPrkDSaHY73Xnh6yBegiZaVi9p/CajnJUSlGWCGjOIu4UdOqotZwL8HikNFJRN6RgGASoqwQzd/I97/CQwI5zlOnzK4jm7fsJRacxMpsEpqZ2Yba0i/6cNSpu9HDquitKCYougrBTY5rhaHzziGpgVswAo0zzcFbMJDXOyYck2UowNK2aCwRzJYM1NNaJ4eyDXwdmzThx34vfPmyevvyyGtYMeoceohWL0Ap2gd6iH+ohFX6Pv0c/oV+1b7U/tb30511q0BA/RRtV3/wHT1xkI</latexit><latexit sha1_base64="w2TBmwlAQZj/vRWi63pfItSWfq0=">AAADT3icbVJdixMxFM20fuyuH9vVR1+CRWhZt3REUJCFFR/0scJ2d7HpDpn0ThuaZIYko9QQ8Bf5XwRffNDf4JtvYqYt3bbrhYHDOSf3JHduWghubLf7I6rVb9y8dXtnd+/O3Xv39xsHD85MXmoGfZaLXF+k1IDgCvqWWwEXhQYqUwHn6fRNpZ9/BG14rk7trIChpGPFM86oDVRyEL0lkisu+WdIHElFCY7YCViKCVeYnFbQe0wEZJY4TDQ302AcUT0dawDlCv80sDByhIpiUnlb6238K0w+Tah1PZ84R7TEeWrCEeXb+LjKyFax20lVz0xT5mJ/1Xwut1a88sSUMnH8OPaXKkSL3JitC3xIeBsf4bWUdnJ4ufkGHx42ntj2Br1KCbrHh+sd8MJPrkDSaHY73Xnh6yBegiZaVi9p/CajnJUSlGWCGjOIu4UdOqotZwL8HikNFJRN6RgGASoqwQzd/I97/CQwI5zlOnzK4jm7fsJRacxMpsEpqZ2Yba0i/6cNSpu9HDquitKCYougrBTY5rhaHzziGpgVswAo0zzcFbMJDXOyYck2UowNK2aCwRzJYM1NNaJ4eyDXwdmzThx34vfPmyevvyyGtYMeoceohWL0Ap2gd6iH+ohFX6Pv0c/oV+1b7U/tb30511q0BA/RRtV3/wHT1xkI</latexit>

Page 26: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Outline

• Worst-case subpopulation performance

• Statistics: minimax optimality

• Experiments

• Overview of broader research agendas

Page 27: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Worst case over distributions P

Minimax (excess) risk

Best case over procedures

A Principle: Minimax

b✓ : Zn ! ⇥<latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit>

minb✓

maxPobs2P

⇢EPobs [Rp,↵(b✓(Zn

1 );Pobs)]�min✓2⇥

Rp,↵(✓;Pobs)

<latexit sha1_base64="q942JknRJQunJ+gBJrtQG9l6Pbg=">AAAC9nicbVJNi9RAEO3Er3X8GvXopXEQZsFdkkVQ8LIigscRdnYXp2Po9HQmzXR3QndFd2gCHvRveBOv/h1/hf4EO8kcZnanoOHx6lW9oqqzSgoLUfQnCK9dv3Hz1t7twZ279+4/GD58dGrL2jA+ZaUszXlGLZdC8ykIkPy8MpyqTPKzbPm2zZ995saKUp/AquKJogstcsEoeCod/iWZrLkjSujUkS8FBUeg4ECbpsGYGD53mCh6kbqJzxuFy8w2mAjtWeYmrYpIngNxA/JuS9TMfLmwy9RVzzGhsipoM167bfqMP6bxJ73fvO7dNjvsJ/gAr0frxL3xSVe3q3sn2tVo4MWLAkiTDkfRYdQFvgriNRihdUzS4T8yL1mtuAYmqbWzOKogcdSAYJI3A1JbXlG2pAs+81BTxW3iusM0+Jln5jgvjX8acMduVjiqrF2pzCsVhcJezrXkrtyshvxV4oSuauCa9UZ5LTGUuL0yngvDGciVB5QZ4WfFrKCGMvB/YcvFgv8J1gvsgfLS0rYrii8v5Co4PTqMPf7wYnT85mu/rD30BD1FYxSjl+gYvUcTNEUsmAYu+BZ8Dy/CH+HP8FcvDYP1gh+jrQh//wdZZPRk</latexit><latexit sha1_base64="q942JknRJQunJ+gBJrtQG9l6Pbg=">AAAC9nicbVJNi9RAEO3Er3X8GvXopXEQZsFdkkVQ8LIigscRdnYXp2Po9HQmzXR3QndFd2gCHvRveBOv/h1/hf4EO8kcZnanoOHx6lW9oqqzSgoLUfQnCK9dv3Hz1t7twZ279+4/GD58dGrL2jA+ZaUszXlGLZdC8ykIkPy8MpyqTPKzbPm2zZ995saKUp/AquKJogstcsEoeCod/iWZrLkjSujUkS8FBUeg4ECbpsGYGD53mCh6kbqJzxuFy8w2mAjtWeYmrYpIngNxA/JuS9TMfLmwy9RVzzGhsipoM167bfqMP6bxJ73fvO7dNjvsJ/gAr0frxL3xSVe3q3sn2tVo4MWLAkiTDkfRYdQFvgriNRihdUzS4T8yL1mtuAYmqbWzOKogcdSAYJI3A1JbXlG2pAs+81BTxW3iusM0+Jln5jgvjX8acMduVjiqrF2pzCsVhcJezrXkrtyshvxV4oSuauCa9UZ5LTGUuL0yngvDGciVB5QZ4WfFrKCGMvB/YcvFgv8J1gvsgfLS0rYrii8v5Co4PTqMPf7wYnT85mu/rD30BD1FYxSjl+gYvUcTNEUsmAYu+BZ8Dy/CH+HP8FcvDYP1gh+jrQh//wdZZPRk</latexit><latexit sha1_base64="q942JknRJQunJ+gBJrtQG9l6Pbg=">AAAC9nicbVJNi9RAEO3Er3X8GvXopXEQZsFdkkVQ8LIigscRdnYXp2Po9HQmzXR3QndFd2gCHvRveBOv/h1/hf4EO8kcZnanoOHx6lW9oqqzSgoLUfQnCK9dv3Hz1t7twZ279+4/GD58dGrL2jA+ZaUszXlGLZdC8ykIkPy8MpyqTPKzbPm2zZ995saKUp/AquKJogstcsEoeCod/iWZrLkjSujUkS8FBUeg4ECbpsGYGD53mCh6kbqJzxuFy8w2mAjtWeYmrYpIngNxA/JuS9TMfLmwy9RVzzGhsipoM167bfqMP6bxJ73fvO7dNjvsJ/gAr0frxL3xSVe3q3sn2tVo4MWLAkiTDkfRYdQFvgriNRihdUzS4T8yL1mtuAYmqbWzOKogcdSAYJI3A1JbXlG2pAs+81BTxW3iusM0+Jln5jgvjX8acMduVjiqrF2pzCsVhcJezrXkrtyshvxV4oSuauCa9UZ5LTGUuL0yngvDGciVB5QZ4WfFrKCGMvB/YcvFgv8J1gvsgfLS0rYrii8v5Co4PTqMPf7wYnT85mu/rD30BD1FYxSjl+gYvUcTNEUsmAYu+BZ8Dy/CH+HP8FcvDYP1gh+jrQh//wdZZPRk</latexit><latexit sha1_base64="q942JknRJQunJ+gBJrtQG9l6Pbg=">AAAC9nicbVJNi9RAEO3Er3X8GvXopXEQZsFdkkVQ8LIigscRdnYXp2Po9HQmzXR3QndFd2gCHvRveBOv/h1/hf4EO8kcZnanoOHx6lW9oqqzSgoLUfQnCK9dv3Hz1t7twZ279+4/GD58dGrL2jA+ZaUszXlGLZdC8ykIkPy8MpyqTPKzbPm2zZ995saKUp/AquKJogstcsEoeCod/iWZrLkjSujUkS8FBUeg4ECbpsGYGD53mCh6kbqJzxuFy8w2mAjtWeYmrYpIngNxA/JuS9TMfLmwy9RVzzGhsipoM167bfqMP6bxJ73fvO7dNjvsJ/gAr0frxL3xSVe3q3sn2tVo4MWLAkiTDkfRYdQFvgriNRihdUzS4T8yL1mtuAYmqbWzOKogcdSAYJI3A1JbXlG2pAs+81BTxW3iusM0+Jln5jgvjX8acMduVjiqrF2pzCsVhcJezrXkrtyshvxV4oSuauCa9UZ5LTGUuL0yngvDGciVB5QZ4WfFrKCGMvB/YcvFgv8J1gvsgfLS0rYrii8v5Co4PTqMPf7wYnT85mu/rD30BD1FYxSjl+gYvUcTNEUsmAYu+BZ8Dy/CH+HP8FcvDYP1gh+jrQh//wdZZPRk</latexit>

[Wald 39, von Neumann 28]:

1. We choose procedure , nature chooses2. Receive data i.i.d. from , makes decision

b✓<latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit>

b✓<latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit><latexit sha1_base64="95oKg9zRDU0wrk/ST2K4nltQHR0=">AAACG3icbZDLSsNAFIYn9VbrLV52boJFcGNJRNBlxY3LCvYCTSmT6Wk7dJIJMydKDQVfRHCrr+FO3LrwLXwEJ20XtvXAwM///8M5fEEsuEbX/bZyS8srq2v59cLG5tb2jr27V9MyUQyqTAqpGgHVIHgEVeQooBEroGEgoB4MrrO8fg9Kcxnd4TCGVkh7Ee9yRtFYbfvAD0QCqf/Qp5j62Aeko1HbLroldzzOovCmokimU2nbP35HsiSECJmgWjc9N8ZWShVyJmBU8BMNMWUD2oOmkRENQbfS8fUj59g4HacrlXkROmP374+UhloPw8A0Q4p9PZ9l5n9ZM8HuZSvlUZwgRGyyqJsIB6WToXA6XAFDMTSCMsXNrQ7rU0UZGmAzWzQaXNoU9GloqlJniLx5IIuidlbyjL49L5avniaw8uSQHJET4pELUiY3pEKqhJFH8kJeyZv1bL1bH9bnpJqzpoD3ycxYX79HTKOG</latexit>

Pobs<latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit>

Pobs<latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit><latexit sha1_base64="5J4YEv6lHaS2KohKTMcoI+JzW5Q=">AAACGHicbZDLSsNAGIUnXmu9pbp0M1gEN5ZEBF1W3LisYC/QhDCZTNqhM5kwM1FKCPgcglt9DXfi1p1v4SM4abuwrT8MHM45w//zhSmjSjvOt7Wyura+sVnZqm7v7O7t27WDjhKZxKSNBROyFyJFGE1IW1PNSC+VBPGQkW44uinz7gORiorkXo9T4nM0SGhMMdLGCuyaJ0mUt4LckxyKUBVFYNedhjMZuCzcmaiD2bQC+8eLBM44STRmSKm+66Taz5HUFDNSVL1MkRThERqQvpEJ4kT5+eT0Ap4YJ4KxkOYlGk7cvz9yxJUa89A0OdJDtZiV5n9ZP9PxlZ/TJM00SfB0UZwxqAUsOcCISoI1GxuBsKTmVoiHSCKsDa25LUobVsoU1Bk3VaFKRO4ikGXROW+4Rt9d1JvXT1NYFXAEjsEpcMElaIJb0AJtgMEjeAGv4M16tt6tD+tzWl2xZoAPwdxYX78VcKHH</latexit>

Zi<latexit sha1_base64="m7AJ0mJZHkuKVJZ4MIL/t3ITQsQ=">AAACCHicbVDLSgMxFL3js9ZX1aWbYBHcWGZE0GXFjcuK9oHtUDJppg3NY0gyQhkKrgW3+hvuxK1/4V/4CaaPhW09EDicc8K990QJZ8b6/re3tLyyurae28hvbm3v7Bb29mtGpZrQKlFc6UaEDeVM0qplltNGoikWEaf1qH898uuPVBum5L0dJDQUuCtZzAi2Trp7aLN2oeiX/DHQIgmmpAhTVNqFn1ZHkVRQaQnHxjQDP7FhhrVlhNNhvpUammDSx13adFRiQU2YjVcdomOndFCstHvSorH690eGhTEDEbmkwLZn5r2R+J/XTG18GWZMJqmlkkwGxSlHVqHR3ajDNCWWDxzBRDO3KyI9rDGxrp2ZKca6bowLmFPhosoMXUXBfCGLpHZWChy/PS+Wr54mZeXgEI7gBAK4gDLcQAWqQKALL/AKb96z9+59eJ+T6JI3LfgAZuB9/QIPNJtX</latexit><latexit sha1_base64="m7AJ0mJZHkuKVJZ4MIL/t3ITQsQ=">AAACCHicbVDLSgMxFL3js9ZX1aWbYBHcWGZE0GXFjcuK9oHtUDJppg3NY0gyQhkKrgW3+hvuxK1/4V/4CaaPhW09EDicc8K990QJZ8b6/re3tLyyurae28hvbm3v7Bb29mtGpZrQKlFc6UaEDeVM0qplltNGoikWEaf1qH898uuPVBum5L0dJDQUuCtZzAi2Trp7aLN2oeiX/DHQIgmmpAhTVNqFn1ZHkVRQaQnHxjQDP7FhhrVlhNNhvpUammDSx13adFRiQU2YjVcdomOndFCstHvSorH690eGhTEDEbmkwLZn5r2R+J/XTG18GWZMJqmlkkwGxSlHVqHR3ajDNCWWDxzBRDO3KyI9rDGxrp2ZKca6bowLmFPhosoMXUXBfCGLpHZWChy/PS+Wr54mZeXgEI7gBAK4gDLcQAWqQKALL/AKb96z9+59eJ+T6JI3LfgAZuB9/QIPNJtX</latexit><latexit sha1_base64="m7AJ0mJZHkuKVJZ4MIL/t3ITQsQ=">AAACCHicbVDLSgMxFL3js9ZX1aWbYBHcWGZE0GXFjcuK9oHtUDJppg3NY0gyQhkKrgW3+hvuxK1/4V/4CaaPhW09EDicc8K990QJZ8b6/re3tLyyurae28hvbm3v7Bb29mtGpZrQKlFc6UaEDeVM0qplltNGoikWEaf1qH898uuPVBum5L0dJDQUuCtZzAi2Trp7aLN2oeiX/DHQIgmmpAhTVNqFn1ZHkVRQaQnHxjQDP7FhhrVlhNNhvpUammDSx13adFRiQU2YjVcdomOndFCstHvSorH690eGhTEDEbmkwLZn5r2R+J/XTG18GWZMJqmlkkwGxSlHVqHR3ajDNCWWDxzBRDO3KyI9rDGxrp2ZKca6bowLmFPhosoMXUXBfCGLpHZWChy/PS+Wr54mZeXgEI7gBAK4gDLcQAWqQKALL/AKb96z9+59eJ+T6JI3LfgAZuB9/QIPNJtX</latexit><latexit sha1_base64="m7AJ0mJZHkuKVJZ4MIL/t3ITQsQ=">AAACCHicbVDLSgMxFL3js9ZX1aWbYBHcWGZE0GXFjcuK9oHtUDJppg3NY0gyQhkKrgW3+hvuxK1/4V/4CaaPhW09EDicc8K990QJZ8b6/re3tLyyurae28hvbm3v7Bb29mtGpZrQKlFc6UaEDeVM0qplltNGoikWEaf1qH898uuPVBum5L0dJDQUuCtZzAi2Trp7aLN2oeiX/DHQIgmmpAhTVNqFn1ZHkVRQaQnHxjQDP7FhhrVlhNNhvpUammDSx13adFRiQU2YjVcdomOndFCstHvSorH690eGhTEDEbmkwLZn5r2R+J/XTG18GWZMJqmlkkwGxSlHVqHR3ajDNCWWDxzBRDO3KyI9rDGxrp2ZKca6bowLmFPhosoMXUXBfCGLpHZWChy/PS+Wr54mZeXgEI7gBAK4gDLcQAWqQKALL/AKb96z9+59eJ+T6JI3LfgAZuB9/QIPNJtX</latexit>

Page 28: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Theorem (D. & Namkoong ’18)

Main Result

Two pronged approach

1. Convergence guarantee: find good procedure

2. Lower bound: show no procedure can do better

Worst case over distributions PBest case over procedures b✓ : Zn ! ⇥

<latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit><latexit sha1_base64="Oz9Thqeaj8B+EgxCt7i6XMr0f5E=">AAACMHicbVDLSgMxFM34flt16SZYBF0oMyIoLqTgxqVCH2Knlkx6a4NJZkjuKGWYb/BfBLf6G7oSt678BNPHwloPBA7nnMvNPVEihUXff/cmJqemZ2bn5hcWl5ZXVgtr61Ubp4ZDhccyNlcRsyCFhgoKlHCVGGAqklCL7s56fu0ejBWxLmM3gYZit1q0BWfopGZhN4xkCln40GGYhdgBZHl+QkPFs+v8RtMQYxqWe3KzUPT3/T7oOAmGpEiGuGgWvsNWzFMFGrlk1tYDP8FGxgwKLiFfCFMLCeN37BbqjmqmwDay/kk53XZKi7Zj455G2ld/T2RMWdtVkUsqhh371+uJ/3n1FNvHjUzoJEXQfLConUrq7uz1Q1vCAEfZdYRxI9xfKe8wwzi6Fke2WHQdWhewe8pFY5u7ioK/hYyT6sF+4PjlYbF0OixrjmySLbJDAnJESuScXJAK4eSRPJMX8uo9eW/eh/c5iE54w5kNMgLv6wdYZKtK</latexit>

: parametric rate, 1 p 2<latexit sha1_base64="jNEm5TFZT1Q7no59muk9aqhagYg=">AAACEnicbZDLSgMxFIbPeK31VnXpJlgEN5aZIuiy4sZlBXuBdiiZNNOG5jImGaGUgg8huNXXcCdufQHfwkcwnXZhWw8k/PznP5zDFyWcGev7397K6tr6xmZuK7+9s7u3Xzg4rBuVakJrRHGlmxE2lDNJa5ZZTpuJplhEnDaiwc2k33ik2jAl7+0woaHAPcliRrB1VhigNqcoyf5yp1D0S35WaFkEM1GEWVU7hZ92V5FUUGkJx8a0Aj+x4Qhrywin43w7NTTBZIB7tOWkxIKacJQdPUanzumiWGn3pEWZ+3dihIUxQxG5pMC2bxZ7E/O/Xiu18VU4YjJJLZVkuihOObIKTQigLtOUWD50AhPN3K2I9LHGxDpOc1uMdZSMC5hz4aLKjB2iYBHIsqiXS4HTdxfFyvXTFFYOjuEEziCAS6jALVShBgQe4AVe4c179t69D+9zGl3xZoCPYK68r1+97p5G</latexit><latexit sha1_base64="jNEm5TFZT1Q7no59muk9aqhagYg=">AAACEnicbZDLSgMxFIbPeK31VnXpJlgEN5aZIuiy4sZlBXuBdiiZNNOG5jImGaGUgg8huNXXcCdufQHfwkcwnXZhWw8k/PznP5zDFyWcGev7397K6tr6xmZuK7+9s7u3Xzg4rBuVakJrRHGlmxE2lDNJa5ZZTpuJplhEnDaiwc2k33ik2jAl7+0woaHAPcliRrB1VhigNqcoyf5yp1D0S35WaFkEM1GEWVU7hZ92V5FUUGkJx8a0Aj+x4Qhrywin43w7NTTBZIB7tOWkxIKacJQdPUanzumiWGn3pEWZ+3dihIUxQxG5pMC2bxZ7E/O/Xiu18VU4YjJJLZVkuihOObIKTQigLtOUWD50AhPN3K2I9LHGxDpOc1uMdZSMC5hz4aLKjB2iYBHIsqiXS4HTdxfFyvXTFFYOjuEEziCAS6jALVShBgQe4AVe4c179t69D+9zGl3xZoCPYK68r1+97p5G</latexit><latexit sha1_base64="jNEm5TFZT1Q7no59muk9aqhagYg=">AAACEnicbZDLSgMxFIbPeK31VnXpJlgEN5aZIuiy4sZlBXuBdiiZNNOG5jImGaGUgg8huNXXcCdufQHfwkcwnXZhWw8k/PznP5zDFyWcGev7397K6tr6xmZuK7+9s7u3Xzg4rBuVakJrRHGlmxE2lDNJa5ZZTpuJplhEnDaiwc2k33ik2jAl7+0woaHAPcliRrB1VhigNqcoyf5yp1D0S35WaFkEM1GEWVU7hZ92V5FUUGkJx8a0Aj+x4Qhrywin43w7NTTBZIB7tOWkxIKacJQdPUanzumiWGn3pEWZ+3dihIUxQxG5pMC2bxZ7E/O/Xiu18VU4YjJJLZVkuihOObIKTQigLtOUWD50AhPN3K2I9LHGxDpOc1uMdZSMC5hz4aLKjB2iYBHIsqiXS4HTdxfFyvXTFFYOjuEEziCAS6jALVShBgQe4AVe4c179t69D+9zGl3xZoCPYK68r1+97p5G</latexit><latexit sha1_base64="jNEm5TFZT1Q7no59muk9aqhagYg=">AAACEnicbZDLSgMxFIbPeK31VnXpJlgEN5aZIuiy4sZlBXuBdiiZNNOG5jImGaGUgg8huNXXcCdufQHfwkcwnXZhWw8k/PznP5zDFyWcGev7397K6tr6xmZuK7+9s7u3Xzg4rBuVakJrRHGlmxE2lDNJa5ZZTpuJplhEnDaiwc2k33ik2jAl7+0woaHAPcliRrB1VhigNqcoyf5yp1D0S35WaFkEM1GEWVU7hZ92V5FUUGkJx8a0Aj+x4Qhrywin43w7NTTBZIB7tOWkxIKacJQdPUanzumiWGn3pEWZ+3dihIUxQxG5pMC2bxZ7E/O/Xiu18VU4YjJJLZVkuihOObIKTQigLtOUWD50AhPN3K2I9LHGxDpOc1uMdZSMC5hz4aLKjB2iYBHIsqiXS4HTdxfFyvXTFFYOjuEEziCAS6jALVShBgQe4AVe4c179t69D+9zGl3xZoCPYK68r1+97p5G</latexit>

p > 2<latexit sha1_base64="JEVkTQpllzt3ufQys2gUjhqLSoU=">AAACCnicbVDLSgMxFM34rPVVdekmWAQ3lpki6EoqblxWcNpCO5RMmmlD8xiSO0IpBT9AcKu/4U7c+hP+hZ9g+ljY1gOBwzkn3HtPnApuwfe/vZXVtfWNzdxWfntnd2+/cHBYszozlIVUC20aMbFMcMVC4CBYIzWMyFiwety/Hfv1R2Ys1+oBBimLJOkqnnBKwElhiq9xuV0o+iV/ArxMghkpohmq7cJPq6NpJpkCKoi1zcBPIRoSA5wKNsq3MstSQvuky5qOKiKZjYaTZUf41CkdnGjjngI8Uf/+GBJp7UDGLikJ9OyiNxb/85oZJFfRkKs0A6bodFCSCQwajy/HHW4YBTFwhFDD3a6Y9oghFFw/c1MsuHasC9hz6aLajlxFwWIhy6RWLgWO318UKzdP07Jy6BidoDMUoEtUQXeoikJEEUcv6BW9ec/eu/fhfU6jK96s4CM0B+/rF1enm2k=</latexit><latexit sha1_base64="JEVkTQpllzt3ufQys2gUjhqLSoU=">AAACCnicbVDLSgMxFM34rPVVdekmWAQ3lpki6EoqblxWcNpCO5RMmmlD8xiSO0IpBT9AcKu/4U7c+hP+hZ9g+ljY1gOBwzkn3HtPnApuwfe/vZXVtfWNzdxWfntnd2+/cHBYszozlIVUC20aMbFMcMVC4CBYIzWMyFiwety/Hfv1R2Ys1+oBBimLJOkqnnBKwElhiq9xuV0o+iV/ArxMghkpohmq7cJPq6NpJpkCKoi1zcBPIRoSA5wKNsq3MstSQvuky5qOKiKZjYaTZUf41CkdnGjjngI8Uf/+GBJp7UDGLikJ9OyiNxb/85oZJFfRkKs0A6bodFCSCQwajy/HHW4YBTFwhFDD3a6Y9oghFFw/c1MsuHasC9hz6aLajlxFwWIhy6RWLgWO318UKzdP07Jy6BidoDMUoEtUQXeoikJEEUcv6BW9ec/eu/fhfU6jK96s4CM0B+/rF1enm2k=</latexit><latexit sha1_base64="JEVkTQpllzt3ufQys2gUjhqLSoU=">AAACCnicbVDLSgMxFM34rPVVdekmWAQ3lpki6EoqblxWcNpCO5RMmmlD8xiSO0IpBT9AcKu/4U7c+hP+hZ9g+ljY1gOBwzkn3HtPnApuwfe/vZXVtfWNzdxWfntnd2+/cHBYszozlIVUC20aMbFMcMVC4CBYIzWMyFiwety/Hfv1R2Ys1+oBBimLJOkqnnBKwElhiq9xuV0o+iV/ArxMghkpohmq7cJPq6NpJpkCKoi1zcBPIRoSA5wKNsq3MstSQvuky5qOKiKZjYaTZUf41CkdnGjjngI8Uf/+GBJp7UDGLikJ9OyiNxb/85oZJFfRkKs0A6bodFCSCQwajy/HHW4YBTFwhFDD3a6Y9oghFFw/c1MsuHasC9hz6aLajlxFwWIhy6RWLgWO318UKzdP07Jy6BidoDMUoEtUQXeoikJEEUcv6BW9ec/eu/fhfU6jK96s4CM0B+/rF1enm2k=</latexit><latexit sha1_base64="JEVkTQpllzt3ufQys2gUjhqLSoU=">AAACCnicbVDLSgMxFM34rPVVdekmWAQ3lpki6EoqblxWcNpCO5RMmmlD8xiSO0IpBT9AcKu/4U7c+hP+hZ9g+ljY1gOBwzkn3HtPnApuwfe/vZXVtfWNzdxWfntnd2+/cHBYszozlIVUC20aMbFMcMVC4CBYIzWMyFiwety/Hfv1R2Ys1+oBBimLJOkqnnBKwElhiq9xuV0o+iV/ArxMghkpohmq7cJPq6NpJpkCKoi1zcBPIRoSA5wKNsq3MstSQvuky5qOKiKZjYaTZUf41CkdnGjjngI8Uf/+GBJp7UDGLikJ9OyiNxb/85oZJFfRkKs0A6bodFCSCQwajy/HHW4YBTFwhFDD3a6Y9oghFFw/c1MsuHasC9hz6aLajlxFwWIhy6RWLgWO318UKzdP07Jy6BidoDMUoEtUQXeoikJEEUcv6BW9ec/eu/fhfU6jK96s4CM0B+/rF1enm2k=</latexit>

: slower rate

minb✓

maxPobs

nEPobs [Rp,↵(b✓(Zn

1 );Pobs)]�min✓2⇥

Rp,↵(✓;Pobs)o⇣ n� 1

max{p,2}

Page 29: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Outline

• Worst-case subpopulation performance

• Statistics: minimax optimality

• Experiments

• Overview of broader research agendas

Page 30: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• Learn from hard subpopulations during training

• Models with uniform performance

• Trade-off between worst-case subpopulation vs average

ExperimentsExperiment 1

Underrepresented demographic groups

Experiment 2Fundamentally difficult

subpopulations

Page 31: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

• Warfarin is the most widely used blood-thinner worldwide

• Y: therapeutic dosage

• X: demographics, genetic info

• Model: linear- Worked best out of polynomial regression, kernel methods, neural networks,

splines, boosting, bagging [IWPC ’09]

• Loss: squared loss

Experiment 1: Warfarin Dosage

`(✓;X,Y ) = (Y � ✓>X)2<latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit><latexit sha1_base64="6uOIBu6bwrhxNlpf/269e/JvZbc=">AAACE3icbZDJSgNBEIZ74hbjFvXopTEIiajMBEFBlIAXjxHMRiYJPZ2KadKz0F0jhJB38OKrePGgiFcv3nwbO8tBE39o+Piriur6vUgKjbb9bSUWFpeWV5KrqbX1jc2t9PZOWYex4lDioQxV1WMapAighAIlVCMFzPckVLze9aheeQClRRjcYT+Chs/uA9ERnKGxWulDF6TMutgFZBe0ekRrOXpJszV6TCdm08UwotVcM99KZ+wTeyw6D84UMmSqYiv95bZDHvsQIJdM67pjR9gYMIWCSxim3FhDxHiP3UPdYMB80I3B+KYhPTBOm3ZCZV6AdOz+nhgwX+u+75lOn2FXz9ZG5n+1eoyd88ZABFGMEPDJok4sKYZ0FBBtCwUcZd8A40qYv1LeZYpxNDGmTAjO7MnzUM6fOIZvTzOFq2kcSbJH9kmWOOSMFMgNKZIS4eSRPJNX8mY9WS/Wu/UxaU1Y05ld8kfW5w8ONZsa</latexit>

Page 32: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Baseline: Empirical risk minimization

High prediction error on patients with high dosage

Problem: poor tail performance

Goal: Reliable personalized treatments

minimize✓2⇥1

n

nX

i=1

(Yi � ✓>Xi)2

<latexit sha1_base64="QLPzQSfxVSHvcXpfSq6j46C6wgc=">AAACaXicbZBNaxRBEIZ7x6+4fmSjF9FL4yLEQ8JMEMxFCPHiMUI2WdneHXp6a7JN+mPorhE2Tfv78hf8DULITY/2fhxMYkHBy1tvUcVTNUp6zPOfneze/QcPH2087j55+uz5Zm/rxYm3rRMwEFZZN6y4ByUNDFCigmHjgOtKwWl1/nkxP/0OzktrjnHewFjzMyNrKTgmq+xVgTlNtTRSywuIZWAz33ABoWgwMpwBcsqkoex4IW P3B6sdF6GIwUTKfKvLID8VcWLo9rdS0h262pkwtA0dlvL9ZK/s9fPdfFn0rijWok/WdVT2rtnUilaDQaG496Mib3AcuEMpFMQuaz2kH8/5GYySNFyDH4cli0jfJWdKa+tSG6RL99+NwLX3c12lpOY487dnC/N/s1GL9f44SNO0CEasDtWtomjpAiydSgcC1TwJLpxMv1Ix4wkWJvw3rnhM8H0K+B2dotbHhKi4DeSuONnbLZL++qF/cLiGtUHekLdkmxTkIzkgX8gRGRBBLskV+U3+dH5lW9mr7PUqmnXWOy/Jjcr6fwF22b3p</latexit><latexit sha1_base64="QLPzQSfxVSHvcXpfSq6j46C6wgc=">AAACaXicbZBNaxRBEIZ7x6+4fmSjF9FL4yLEQ8JMEMxFCPHiMUI2WdneHXp6a7JN+mPorhE2Tfv78hf8DULITY/2fhxMYkHBy1tvUcVTNUp6zPOfneze/QcPH2087j55+uz5Zm/rxYm3rRMwEFZZN6y4ByUNDFCigmHjgOtKwWl1/nkxP/0OzktrjnHewFjzMyNrKTgmq+xVgTlNtTRSywuIZWAz33ABoWgwMpwBcsqkoex4IW P3B6sdF6GIwUTKfKvLID8VcWLo9rdS0h262pkwtA0dlvL9ZK/s9fPdfFn0rijWok/WdVT2rtnUilaDQaG496Mib3AcuEMpFMQuaz2kH8/5GYySNFyDH4cli0jfJWdKa+tSG6RL99+NwLX3c12lpOY487dnC/N/s1GL9f44SNO0CEasDtWtomjpAiydSgcC1TwJLpxMv1Ix4wkWJvw3rnhM8H0K+B2dotbHhKi4DeSuONnbLZL++qF/cLiGtUHekLdkmxTkIzkgX8gRGRBBLskV+U3+dH5lW9mr7PUqmnXWOy/Jjcr6fwF22b3p</latexit><latexit sha1_base64="QLPzQSfxVSHvcXpfSq6j46C6wgc=">AAACaXicbZBNaxRBEIZ7x6+4fmSjF9FL4yLEQ8JMEMxFCPHiMUI2WdneHXp6a7JN+mPorhE2Tfv78hf8DULITY/2fhxMYkHBy1tvUcVTNUp6zPOfneze/QcPH2087j55+uz5Zm/rxYm3rRMwEFZZN6y4ByUNDFCigmHjgOtKwWl1/nkxP/0OzktrjnHewFjzMyNrKTgmq+xVgTlNtTRSywuIZWAz33ABoWgwMpwBcsqkoex4IW P3B6sdF6GIwUTKfKvLID8VcWLo9rdS0h262pkwtA0dlvL9ZK/s9fPdfFn0rijWok/WdVT2rtnUilaDQaG496Mib3AcuEMpFMQuaz2kH8/5GYySNFyDH4cli0jfJWdKa+tSG6RL99+NwLX3c12lpOY487dnC/N/s1GL9f44SNO0CEasDtWtomjpAiydSgcC1TwJLpxMv1Ix4wkWJvw3rnhM8H0K+B2dotbHhKi4DeSuONnbLZL++qF/cLiGtUHekLdkmxTkIzkgX8gRGRBBLskV+U3+dH5lW9mr7PUqmnXWOy/Jjcr6fwF22b3p</latexit><latexit sha1_base64="QLPzQSfxVSHvcXpfSq6j46C6wgc=">AAACaXicbZBNaxRBEIZ7x6+4fmSjF9FL4yLEQ8JMEMxFCPHiMUI2WdneHXp6a7JN+mPorhE2Tfv78hf8DULITY/2fhxMYkHBy1tvUcVTNUp6zPOfneze/QcPH2087j55+uz5Zm/rxYm3rRMwEFZZN6y4ByUNDFCigmHjgOtKwWl1/nkxP/0OzktrjnHewFjzMyNrKTgmq+xVgTlNtTRSywuIZWAz33ABoWgwMpwBcsqkoex4IW P3B6sdF6GIwUTKfKvLID8VcWLo9rdS0h262pkwtA0dlvL9ZK/s9fPdfFn0rijWok/WdVT2rtnUilaDQaG496Mib3AcuEMpFMQuaz2kH8/5GYySNFyDH4cli0jfJWdKa+tSG6RL99+NwLX3c12lpOY487dnC/N/s1GL9f44SNO0CEasDtWtomjpAiydSgcC1TwJLpxMv1Ix4wkWJvw3rnhM8H0K+B2dotbHhKi4DeSuONnbLZL++qF/cLiGtUHekLdkmxTkIzkgX8gRGRBBLskV+U3+dH5lW9mr7PUqmnXWOy/Jjcr6fwF22b3p</latexit>

minimize✓2⇥1

n

nX

i=1

`(✓;Xi, Yi)<latexit sha1_base64="HNkCubbnLLdlSx9R3aUoDN+VNpU=">AAACZ3icbZDbahRBEIZ7x1NcD1kVRPCmdREiaJgRQUGEoDdeRsgmKzvr2NNbk23Sh6GrRlib9vF8Bx9B8MIbvbb3cGESCwp+/vqLKr661Qopz7/3sgsXL12+snW1f+36jZvbg1u3D9F1XsJIOu38uBYIWlkYkSIN49aDMLWGo/rk7XJ+9Bk8KmcPaNHC1IhjqxolBSWrGnwKpTfcKKuM+gKxCuUcWyEhFC3FkuZAgpfK8vJgKW P/a9l4IUMRg428xM5UQb0u4seU0A5xZ73yio8r9YR/qNTjajDMd/NV8fOi2Igh29R+NfhVzpzsDFiSWiBOirylaRCelNQQ+2WHkD48EccwSdIKAzgNKxKRP0rOjDfOp7bEV+6/G0EYxIWpU9IImuPZ2dL832zSUfNyGpRtOwIr14eaTnNyfImVz5QHSXqRhJBepV+5nIuEihL8U1eQEnpMAXxqUtRhTIiKs0DOi8Nnu0XS758P995sYG2x++wh22EFe8H22Du2z0ZMsm/sJ/vN/vR+ZNvZ3ezeOpr1Njt32KnKHvwFSKC9bw==</latexit><latexit sha1_base64="HNkCubbnLLdlSx9R3aUoDN+VNpU=">AAACZ3icbZDbahRBEIZ7x1NcD1kVRPCmdREiaJgRQUGEoDdeRsgmKzvr2NNbk23Sh6GrRlib9vF8Bx9B8MIbvbb3cGESCwp+/vqLKr661Qopz7/3sgsXL12+snW1f+36jZvbg1u3D9F1XsJIOu38uBYIWlkYkSIN49aDMLWGo/rk7XJ+9Bk8KmcPaNHC1IhjqxolBSWrGnwKpTfcKKuM+gKxCuUcWyEhFC3FkuZAgpfK8vJgKW P/a9l4IUMRg428xM5UQb0u4seU0A5xZ73yio8r9YR/qNTjajDMd/NV8fOi2Igh29R+NfhVzpzsDFiSWiBOirylaRCelNQQ+2WHkD48EccwSdIKAzgNKxKRP0rOjDfOp7bEV+6/G0EYxIWpU9IImuPZ2dL832zSUfNyGpRtOwIr14eaTnNyfImVz5QHSXqRhJBepV+5nIuEihL8U1eQEnpMAXxqUtRhTIiKs0DOi8Nnu0XS758P995sYG2x++wh22EFe8H22Du2z0ZMsm/sJ/vN/vR+ZNvZ3ezeOpr1Njt32KnKHvwFSKC9bw==</latexit><latexit sha1_base64="HNkCubbnLLdlSx9R3aUoDN+VNpU=">AAACZ3icbZDbahRBEIZ7x1NcD1kVRPCmdREiaJgRQUGEoDdeRsgmKzvr2NNbk23Sh6GrRlib9vF8Bx9B8MIbvbb3cGESCwp+/vqLKr661Qopz7/3sgsXL12+snW1f+36jZvbg1u3D9F1XsJIOu38uBYIWlkYkSIN49aDMLWGo/rk7XJ+9Bk8KmcPaNHC1IhjqxolBSWrGnwKpTfcKKuM+gKxCuUcWyEhFC3FkuZAgpfK8vJgKW P/a9l4IUMRg428xM5UQb0u4seU0A5xZ73yio8r9YR/qNTjajDMd/NV8fOi2Igh29R+NfhVzpzsDFiSWiBOirylaRCelNQQ+2WHkD48EccwSdIKAzgNKxKRP0rOjDfOp7bEV+6/G0EYxIWpU9IImuPZ2dL832zSUfNyGpRtOwIr14eaTnNyfImVz5QHSXqRhJBepV+5nIuEihL8U1eQEnpMAXxqUtRhTIiKs0DOi8Nnu0XS758P995sYG2x++wh22EFe8H22Du2z0ZMsm/sJ/vN/vR+ZNvZ3ezeOpr1Njt32KnKHvwFSKC9bw==</latexit><latexit sha1_base64="HNkCubbnLLdlSx9R3aUoDN+VNpU=">AAACZ3icbZDbahRBEIZ7x1NcD1kVRPCmdREiaJgRQUGEoDdeRsgmKzvr2NNbk23Sh6GrRlib9vF8Bx9B8MIbvbb3cGESCwp+/vqLKr661Qopz7/3sgsXL12+snW1f+36jZvbg1u3D9F1XsJIOu38uBYIWlkYkSIN49aDMLWGo/rk7XJ+9Bk8KmcPaNHC1IhjqxolBSWrGnwKpTfcKKuM+gKxCuUcWyEhFC3FkuZAgpfK8vJgKW P/a9l4IUMRg428xM5UQb0u4seU0A5xZ73yio8r9YR/qNTjajDMd/NV8fOi2Igh29R+NfhVzpzsDFiSWiBOirylaRCelNQQ+2WHkD48EccwSdIKAzgNKxKRP0rOjDfOp7bEV+6/G0EYxIWpU9IImuPZ2dL832zSUfNyGpRtOwIr14eaTnNyfImVz5QHSXqRhJBepV+5nIuEihL8U1eQEnpMAXxqUtRhTIiKs0DOi8Nnu0XS758P995sYG2x++wh22EFe8H22Du2z0ZMsm/sJ/vN/vR+ZNvZ3ezeOpr1Njt32KnKHvwFSKC9bw==</latexit>

minimize✓2⇥ EPobs [`(✓;X,Y )]<latexit sha1_base64="oI8DDkyEnJas0pqGFS3FLOZb4VA=">AAACXXicbZBNaxRBEIZ7R6NxjXHVgwcvjYsQQcOMCBFyCYrgcYXsZmV7GHp6a7NN+mPoqglshskvyx/xJHjVgz/B3o+DSSxoeHjrLar6LSujkdL0eye5c3fr3v3tB92HO492H/eePB2hr4OCofLGh3EpEYx2MCRNBsZVAGlLAyfl2adl/+QcAmrvjmlRQW7lqdMzrSRFqeiNGhEst9ppqy+gLRoxx0oqaLKKWkFzIMmFdlwcL7HtXorPRTMoVlO+xLblE2E84t7ae8jHb/i313nR66f76ar4bcg20GebGhS9P2LqVW3BkTIScZKlFeWNDKSVgbYraoR415k8hUlEJy1g3qz+3/JXUZnymQ/xOeIr9d+JRlrEhS2j00qa483eUvxfb1LT7EPeaFfVBE6tF81qw8nzZZh8qgMoMosIUgUdb+VqLoNUFCO/tgUpBo7RgG9ttHpsY0TZzUBuw+jdfhb56/v+0cdNWNvsBXvJ9ljGDtgR+8IGbMgUu2I/2S/2u/Mj2Up2kt21NelsZp6xa5U8/wsFrrkf</latexit><latexit sha1_base64="oI8DDkyEnJas0pqGFS3FLOZb4VA=">AAACXXicbZBNaxRBEIZ7R6NxjXHVgwcvjYsQQcOMCBFyCYrgcYXsZmV7GHp6a7NN+mPoqglshskvyx/xJHjVgz/B3o+DSSxoeHjrLar6LSujkdL0eye5c3fr3v3tB92HO492H/eePB2hr4OCofLGh3EpEYx2MCRNBsZVAGlLAyfl2adl/+QcAmrvjmlRQW7lqdMzrSRFqeiNGhEst9ppqy+gLRoxx0oqaLKKWkFzIMmFdlwcL7HtXorPRTMoVlO+xLblE2E84t7ae8jHb/i313nR66f76ar4bcg20GebGhS9P2LqVW3BkTIScZKlFeWNDKSVgbYraoR415k8hUlEJy1g3qz+3/JXUZnymQ/xOeIr9d+JRlrEhS2j00qa483eUvxfb1LT7EPeaFfVBE6tF81qw8nzZZh8qgMoMosIUgUdb+VqLoNUFCO/tgUpBo7RgG9ttHpsY0TZzUBuw+jdfhb56/v+0cdNWNvsBXvJ9ljGDtgR+8IGbMgUu2I/2S/2u/Mj2Up2kt21NelsZp6xa5U8/wsFrrkf</latexit><latexit sha1_base64="oI8DDkyEnJas0pqGFS3FLOZb4VA=">AAACXXicbZBNaxRBEIZ7R6NxjXHVgwcvjYsQQcOMCBFyCYrgcYXsZmV7GHp6a7NN+mPoqglshskvyx/xJHjVgz/B3o+DSSxoeHjrLar6LSujkdL0eye5c3fr3v3tB92HO492H/eePB2hr4OCofLGh3EpEYx2MCRNBsZVAGlLAyfl2adl/+QcAmrvjmlRQW7lqdMzrSRFqeiNGhEst9ppqy+gLRoxx0oqaLKKWkFzIMmFdlwcL7HtXorPRTMoVlO+xLblE2E84t7ae8jHb/i313nR66f76ar4bcg20GebGhS9P2LqVW3BkTIScZKlFeWNDKSVgbYraoR415k8hUlEJy1g3qz+3/JXUZnymQ/xOeIr9d+JRlrEhS2j00qa483eUvxfb1LT7EPeaFfVBE6tF81qw8nzZZh8qgMoMosIUgUdb+VqLoNUFCO/tgUpBo7RgG9ttHpsY0TZzUBuw+jdfhb56/v+0cdNWNvsBXvJ9ljGDtgR+8IGbMgUu2I/2S/2u/Mj2Up2kt21NelsZp6xa5U8/wsFrrkf</latexit><latexit sha1_base64="oI8DDkyEnJas0pqGFS3FLOZb4VA=">AAACXXicbZBNaxRBEIZ7R6NxjXHVgwcvjYsQQcOMCBFyCYrgcYXsZmV7GHp6a7NN+mPoqglshskvyx/xJHjVgz/B3o+DSSxoeHjrLar6LSujkdL0eye5c3fr3v3tB92HO492H/eePB2hr4OCofLGh3EpEYx2MCRNBsZVAGlLAyfl2adl/+QcAmrvjmlRQW7lqdMzrSRFqeiNGhEst9ppqy+gLRoxx0oqaLKKWkFzIMmFdlwcL7HtXorPRTMoVlO+xLblE2E84t7ae8jHb/i313nR66f76ar4bcg20GebGhS9P2LqVW3BkTIScZKlFeWNDKSVgbYraoR415k8hUlEJy1g3qz+3/JXUZnymQ/xOeIr9d+JRlrEhS2j00qa483eUvxfb1LT7EPeaFfVBE6tF81qw8nzZZh8qgMoMosIUgUdb+VqLoNUFCO/tgUpBo7RgG9ttHpsY0TZzUBuw+jdfhb56/v+0cdNWNvsBXvJ9ljGDtgR+8IGbMgUu2I/2S/2u/Mj2Up2kt21NelsZp6xa5U8/wsFrrkf</latexit>

Empirical risk minimization (ERM)

Page 33: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

High Warfarin Dosage (>49mg)

Average error for ERM = .997

Page 34: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

High Warfarin Dosage (>49mg)

Improved performance on hard subpopulation, slight deterioration in average-caseTakeaway:b✓2,↵,n 2 argmin

✓2⇥

⇢R2,↵(✓; bPobs,n) := inf

⇢1

⇣E bPobs

(`(✓;Z)� ⌘)2+

⌘ 12+ ⌘

��

<latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit>

Page 35: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Experiment 2: Fine-grained recognition• Task: classify image of dog to breed (120 classes)• Kernel features

Stanford Dogs Dataset [Khosla et al. ’11]

No underrepresentation: same number of images per class

Page 36: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

ERM error rate

Hard Easy

BIG gap in performance even when no underrepresentation

Page 37: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

1.0 (ERM) .99 .90 .60 .20 .07

Variation in error over 120 class

lower bound on minority proportion ↵<latexit sha1_base64="KARx87OSnzhq7kxVWik6aFuGE3k=">AAACOHicbZDLSgMxGIUzXmu9jbp0EyyCG8uMCLqsuHFZwV6gLSWTSdvQ3Egy6jBUfAvfRXCrz+DOnbgTn8D0srCtPwQO55zkJ1+kGDU2CN69hcWl5ZXV3Fp+fWNza9vf2a0amWhMKlgyqesRMoRRQSqWWkbqShPEI0ZqUf9ymNduiTZUihubKtLiqCtoh2JkndX2wyaP5H3G5B3RMJKJiKEUkFMhNbUpVFoqqYfVwUMTMdVD+bZfCIrBaOC8CCeiACZTbvvfzVjihBNhMUPGNMJA2VaG3LOYkUG+mRiiEO6jLmk4KRAnppWNvjaAh86JYUdqd4SFI/fvjQxxY1IeuSZHtmdms6H5X9ZIbOe8lVGhEksEHi/qJAxaCYecYEw1wZalTiDsWFAMcQ9phK2jObXFWMfSuII55q4qzcAhCmeBzIvqSTF0+vq0ULp4HMPKgX1wAI5ACM5ACVyBMqgADJ7AC3gFb96z9+F9el/j6oI3AbwHpsb7+QXsFq+b</latexit><latexit sha1_base64="KARx87OSnzhq7kxVWik6aFuGE3k=">AAACOHicbZDLSgMxGIUzXmu9jbp0EyyCG8uMCLqsuHFZwV6gLSWTSdvQ3Egy6jBUfAvfRXCrz+DOnbgTn8D0srCtPwQO55zkJ1+kGDU2CN69hcWl5ZXV3Fp+fWNza9vf2a0amWhMKlgyqesRMoRRQSqWWkbqShPEI0ZqUf9ymNduiTZUihubKtLiqCtoh2JkndX2wyaP5H3G5B3RMJKJiKEUkFMhNbUpVFoqqYfVwUMTMdVD+bZfCIrBaOC8CCeiACZTbvvfzVjihBNhMUPGNMJA2VaG3LOYkUG+mRiiEO6jLmk4KRAnppWNvjaAh86JYUdqd4SFI/fvjQxxY1IeuSZHtmdms6H5X9ZIbOe8lVGhEksEHi/qJAxaCYecYEw1wZalTiDsWFAMcQ9phK2jObXFWMfSuII55q4qzcAhCmeBzIvqSTF0+vq0ULp4HMPKgX1wAI5ACM5ACVyBMqgADJ7AC3gFb96z9+F9el/j6oI3AbwHpsb7+QXsFq+b</latexit><latexit sha1_base64="KARx87OSnzhq7kxVWik6aFuGE3k=">AAACOHicbZDLSgMxGIUzXmu9jbp0EyyCG8uMCLqsuHFZwV6gLSWTSdvQ3Egy6jBUfAvfRXCrz+DOnbgTn8D0srCtPwQO55zkJ1+kGDU2CN69hcWl5ZXV3Fp+fWNza9vf2a0amWhMKlgyqesRMoRRQSqWWkbqShPEI0ZqUf9ymNduiTZUihubKtLiqCtoh2JkndX2wyaP5H3G5B3RMJKJiKEUkFMhNbUpVFoqqYfVwUMTMdVD+bZfCIrBaOC8CCeiACZTbvvfzVjihBNhMUPGNMJA2VaG3LOYkUG+mRiiEO6jLmk4KRAnppWNvjaAh86JYUdqd4SFI/fvjQxxY1IeuSZHtmdms6H5X9ZIbOe8lVGhEksEHi/qJAxaCYecYEw1wZalTiDsWFAMcQ9phK2jObXFWMfSuII55q4qzcAhCmeBzIvqSTF0+vq0ULp4HMPKgX1wAI5ACM5ACVyBMqgADJ7AC3gFb96z9+F9el/j6oI3AbwHpsb7+QXsFq+b</latexit><latexit sha1_base64="KARx87OSnzhq7kxVWik6aFuGE3k=">AAACOHicbZDLSgMxGIUzXmu9jbp0EyyCG8uMCLqsuHFZwV6gLSWTSdvQ3Egy6jBUfAvfRXCrz+DOnbgTn8D0srCtPwQO55zkJ1+kGDU2CN69hcWl5ZXV3Fp+fWNza9vf2a0amWhMKlgyqesRMoRRQSqWWkbqShPEI0ZqUf9ymNduiTZUihubKtLiqCtoh2JkndX2wyaP5H3G5B3RMJKJiKEUkFMhNbUpVFoqqYfVwUMTMdVD+bZfCIrBaOC8CCeiACZTbvvfzVjihBNhMUPGNMJA2VaG3LOYkUG+mRiiEO6jLmk4KRAnppWNvjaAh86JYUdqd4SFI/fvjQxxY1IeuSZHtmdms6H5X9ZIbOe8lVGhEksEHi/qJAxaCYecYEw1wZalTiDsWFAMcQ9phK2jObXFWMfSuII55q4qzcAhCmeBzIvqSTF0+vq0ULp4HMPKgX1wAI5ACM5ACVyBMqgADJ7AC3gFb96z9+F9el/j6oI3AbwHpsb7+QXsFq+b</latexit>

stan

dard

dev

iatio

n of

clas

s-w

ise e

rror

Page 38: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Worst k-classes

b✓2,↵,n 2 argmin✓2⇥

⇢R2,↵(✓; bPobs,n) := inf

⇢1

⇣E bPobs

(`(✓;Z)� ⌘)2+

⌘ 12+ ⌘

��

<latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit><latexit sha1_base64="UZSaUyPZ5t8AGSGQVfGTSdvdibY=">AAADPHicbVJda9swFJW9ry77SrfHvYiFQULTEofCCnspG4M9ZtC0ZVFqZEWORWTZk+SOIPxL9l8Ge91+wd73NgZ72vOu7axN0wkMR/ece464vlEuhbGDwXfPv3Hz1u07W3db9+4/ePiovf342GSFZnzMMpnp04gaLoXiYyus5Ke55jSNJD+JFq8r/uScayMydWSXOZ+mdK5ELBi1UAq3vX0SyYI78jGh1hGbcEvL0A37mFCZJ7SPVVliIhTc9TwVKnSrhlraMEd1F8gkjy1xLaKFWTQmms9c41SW3fXO8iVuMkcQ54hOcRaZsq/KXovMeMw/tMA6hrRKu+Zc+cWaMheUF8YN3cXkTeguTf95Qq7MjNlMf9/Du7hy74U7Z0N48jyxvbNL8yH47tSCVkOS8gKE7c5gb1AffB0EK9BBqzMK27/JLGNFypVlkhozCQa5nTqqrWCSg3FheE7Zgs75BKCiKTdTV//eEj+HygzHmYZPWVxX1zscTY1ZphEoU2oTs8lVxf9xk8LGB1MnVF5YrlgTFBcS2wxXu4JnQnNm5RIAZVrAWzFLKMzHwkZdSTEW9smAwOymIM1MNaJgcyDXwfFwLwD8br9z+Go1rC30FD1DXRSgF+gQvUUjNEbM++R98b563/zP/g//p/+rkfrequcJunL8P38BiFIOzw==</latexit>

Guarantee uniform performance across dog breedsTakeaway:

↵ = .99<latexit sha1_base64="q7cV5qYhwlx90h2DfJMmqhNuVug=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgmYhBNy4jGA+kBlCT6cnadI/unuEMOQUglu9hjtx6wm8hUewk8zCJBY8KKrq8R4VK0aN9f1vr7C2vrG5Vdwu7ezu7R+UD49aRqYakyaWTOpOjAxhVJCmpZaRjtIE8ZiRdjy6m/rtJ6INleLRjhWJOBoImlCMrJPCEDE1RPAWVmu1XrniV/0Z4CoJclIBORq98k/YlzjlRFjMkDHdwFc2ypC2FDMyKYWpIQrhERqQrqMCcWKibPbzBJ45pQ8Tqd0IC2fq340McWPGPHZJjuzQLHtT8T+vm9rkJsqoUKklAs8PJSmDVsJpAbBPNcGWjR1BWFP3K8RDpBG2rqaFK8a6kowLmAvuotJMXEXBciGrpHVZDRx/uKrUa3lZRXACTsE5CMA1qIN70ABNgIECL+AVvHnP3rv34X3OowUv3zkGC/C+fgEmDZ2G</latexit><latexit sha1_base64="q7cV5qYhwlx90h2DfJMmqhNuVug=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgmYhBNy4jGA+kBlCT6cnadI/unuEMOQUglu9hjtx6wm8hUewk8zCJBY8KKrq8R4VK0aN9f1vr7C2vrG5Vdwu7ezu7R+UD49aRqYakyaWTOpOjAxhVJCmpZaRjtIE8ZiRdjy6m/rtJ6INleLRjhWJOBoImlCMrJPCEDE1RPAWVmu1XrniV/0Z4CoJclIBORq98k/YlzjlRFjMkDHdwFc2ypC2FDMyKYWpIQrhERqQrqMCcWKibPbzBJ45pQ8Tqd0IC2fq340McWPGPHZJjuzQLHtT8T+vm9rkJsqoUKklAs8PJSmDVsJpAbBPNcGWjR1BWFP3K8RDpBG2rqaFK8a6kowLmAvuotJMXEXBciGrpHVZDRx/uKrUa3lZRXACTsE5CMA1qIN70ABNgIECL+AVvHnP3rv34X3OowUv3zkGC/C+fgEmDZ2G</latexit><latexit sha1_base64="q7cV5qYhwlx90h2DfJMmqhNuVug=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgmYhBNy4jGA+kBlCT6cnadI/unuEMOQUglu9hjtx6wm8hUewk8zCJBY8KKrq8R4VK0aN9f1vr7C2vrG5Vdwu7ezu7R+UD49aRqYakyaWTOpOjAxhVJCmpZaRjtIE8ZiRdjy6m/rtJ6INleLRjhWJOBoImlCMrJPCEDE1RPAWVmu1XrniV/0Z4CoJclIBORq98k/YlzjlRFjMkDHdwFc2ypC2FDMyKYWpIQrhERqQrqMCcWKibPbzBJ45pQ8Tqd0IC2fq340McWPGPHZJjuzQLHtT8T+vm9rkJsqoUKklAs8PJSmDVsJpAbBPNcGWjR1BWFP3K8RDpBG2rqaFK8a6kowLmAvuotJMXEXBciGrpHVZDRx/uKrUa3lZRXACTsE5CMA1qIN70ABNgIECL+AVvHnP3rv34X3OowUv3zkGC/C+fgEmDZ2G</latexit><latexit sha1_base64="q7cV5qYhwlx90h2DfJMmqhNuVug=">AAACEXicbVBLSgNBFOyJvxh/UZduGoPgxjAjgmYhBNy4jGA+kBlCT6cnadI/unuEMOQUglu9hjtx6wm8hUewk8zCJBY8KKrq8R4VK0aN9f1vr7C2vrG5Vdwu7ezu7R+UD49aRqYakyaWTOpOjAxhVJCmpZaRjtIE8ZiRdjy6m/rtJ6INleLRjhWJOBoImlCMrJPCEDE1RPAWVmu1XrniV/0Z4CoJclIBORq98k/YlzjlRFjMkDHdwFc2ypC2FDMyKYWpIQrhERqQrqMCcWKibPbzBJ45pQ8Tqd0IC2fq340McWPGPHZJjuzQLHtT8T+vm9rkJsqoUKklAs8PJSmDVsJpAbBPNcGWjR1BWFP3K8RDpBG2rqaFK8a6kowLmAvuotJMXEXBciGrpHVZDRx/uKrUa3lZRXACTsE5CMA1qIN70ABNgIECL+AVvHnP3rv34X3OowUv3zkGC/C+fgEmDZ2G</latexit>

↵ = .90<latexit sha1_base64="Juf151MKjS7kJ22Aqju1OvhbWok=">AAACEnicbVBLSgNBFHzjN8Zf1KWbxiC4McyIoFkIATcuI5gPJEPo6XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbETaUM0lrlllOm7GmWEScNqLh7cRvPFFtmJIPdhTTUOC+ZD1GsHVS2MY8HmB0g0plP98pFP2SPwVaJkFGipCh2in8tLuKJIJKSzg2phX4sQ1TrC0jnI7z7cTQGJMh7tOWoxILasJ0+vQYnTqli3pKu5EWTdW/GykWxoxE5JIC24FZ9Cbif14rsb3rMGUyTiyVZHaol3BkFZo0gLpMU2L5yBFMNHO/IjLAGhPrepq7YqxrybiAORcuqszYVRQsFrJM6helwPH7y2KlnJWVg2M4gTMI4AoqcAdVqAGBR3iBV3jznr1378P7nEVXvGznCObgff0CUD2dkQ==</latexit><latexit sha1_base64="Juf151MKjS7kJ22Aqju1OvhbWok=">AAACEnicbVBLSgNBFHzjN8Zf1KWbxiC4McyIoFkIATcuI5gPJEPo6XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbETaUM0lrlllOm7GmWEScNqLh7cRvPFFtmJIPdhTTUOC+ZD1GsHVS2MY8HmB0g0plP98pFP2SPwVaJkFGipCh2in8tLuKJIJKSzg2phX4sQ1TrC0jnI7z7cTQGJMh7tOWoxILasJ0+vQYnTqli3pKu5EWTdW/GykWxoxE5JIC24FZ9Cbif14rsb3rMGUyTiyVZHaol3BkFZo0gLpMU2L5yBFMNHO/IjLAGhPrepq7YqxrybiAORcuqszYVRQsFrJM6helwPH7y2KlnJWVg2M4gTMI4AoqcAdVqAGBR3iBV3jznr1378P7nEVXvGznCObgff0CUD2dkQ==</latexit><latexit sha1_base64="Juf151MKjS7kJ22Aqju1OvhbWok=">AAACEnicbVBLSgNBFHzjN8Zf1KWbxiC4McyIoFkIATcuI5gPJEPo6XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbETaUM0lrlllOm7GmWEScNqLh7cRvPFFtmJIPdhTTUOC+ZD1GsHVS2MY8HmB0g0plP98pFP2SPwVaJkFGipCh2in8tLuKJIJKSzg2phX4sQ1TrC0jnI7z7cTQGJMh7tOWoxILasJ0+vQYnTqli3pKu5EWTdW/GykWxoxE5JIC24FZ9Cbif14rsb3rMGUyTiyVZHaol3BkFZo0gLpMU2L5yBFMNHO/IjLAGhPrepq7YqxrybiAORcuqszYVRQsFrJM6helwPH7y2KlnJWVg2M4gTMI4AoqcAdVqAGBR3iBV3jznr1378P7nEVXvGznCObgff0CUD2dkQ==</latexit><latexit sha1_base64="Juf151MKjS7kJ22Aqju1OvhbWok=">AAACEnicbVBLSgNBFHzjN8Zf1KWbxiC4McyIoFkIATcuI5gPJEPo6XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbETaUM0lrlllOm7GmWEScNqLh7cRvPFFtmJIPdhTTUOC+ZD1GsHVS2MY8HmB0g0plP98pFP2SPwVaJkFGipCh2in8tLuKJIJKSzg2phX4sQ1TrC0jnI7z7cTQGJMh7tOWoxILasJ0+vQYnTqli3pKu5EWTdW/GykWxoxE5JIC24FZ9Cbif14rsb3rMGUyTiyVZHaol3BkFZo0gLpMU2L5yBFMNHO/IjLAGhPrepq7YqxrybiAORcuqszYVRQsFrJM6helwPH7y2KlnJWVg2M4gTMI4AoqcAdVqAGBR3iBV3jznr1378P7nEVXvGznCObgff0CUD2dkQ==</latexit>

↵ = .60<latexit sha1_base64="OyVw6ZLrc06cHCxa/LbkYfBkrXA=">AAACEnicbVDJSgNBFOxxjXGLevTSGAQvhhkRl4MQ8OIxglkgGUJPpydp0svY/UYIIX8heNXf8CZe/QH/wk+wk8zBJBY8KKrq8R4VJYJb8P1vb2l5ZXVtPbeR39za3tkt7O3XrE4NZVWqhTaNiFgmuGJV4CBYIzGMyEiwetS/Hfv1J2Ys1+oBBgkLJekqHnNKwElhi4ikR/ANLl34+Xah6Jf8CfAiCTJSRBkq7cJPq6NpKpkCKoi1zcBPIBwSA5wKNsq3UssSQvuky5qOKiKZDYeTp0f42CkdHGvjRgGeqH83hkRaO5CRS0oCPTvvjcX/vGYK8VU45CpJgSk6PRSnAoPG4wZwhxtGQQwcIdRw9yumPWIIBdfTzBULriXrAvZUuqi2I1dRMF/IIqmdlQLH78+L5eusrBw6REfoBAXoEpXRHaqgKqLoEb2gV/TmPXvv3of3OY0uednOAZqB9/ULS1Gdjg==</latexit><latexit sha1_base64="OyVw6ZLrc06cHCxa/LbkYfBkrXA=">AAACEnicbVDJSgNBFOxxjXGLevTSGAQvhhkRl4MQ8OIxglkgGUJPpydp0svY/UYIIX8heNXf8CZe/QH/wk+wk8zBJBY8KKrq8R4VJYJb8P1vb2l5ZXVtPbeR39za3tkt7O3XrE4NZVWqhTaNiFgmuGJV4CBYIzGMyEiwetS/Hfv1J2Ys1+oBBgkLJekqHnNKwElhi4ikR/ANLl34+Xah6Jf8CfAiCTJSRBkq7cJPq6NpKpkCKoi1zcBPIBwSA5wKNsq3UssSQvuky5qOKiKZDYeTp0f42CkdHGvjRgGeqH83hkRaO5CRS0oCPTvvjcX/vGYK8VU45CpJgSk6PRSnAoPG4wZwhxtGQQwcIdRw9yumPWIIBdfTzBULriXrAvZUuqi2I1dRMF/IIqmdlQLH78+L5eusrBw6REfoBAXoEpXRHaqgKqLoEb2gV/TmPXvv3of3OY0uednOAZqB9/ULS1Gdjg==</latexit><latexit sha1_base64="OyVw6ZLrc06cHCxa/LbkYfBkrXA=">AAACEnicbVDJSgNBFOxxjXGLevTSGAQvhhkRl4MQ8OIxglkgGUJPpydp0svY/UYIIX8heNXf8CZe/QH/wk+wk8zBJBY8KKrq8R4VJYJb8P1vb2l5ZXVtPbeR39za3tkt7O3XrE4NZVWqhTaNiFgmuGJV4CBYIzGMyEiwetS/Hfv1J2Ys1+oBBgkLJekqHnNKwElhi4ikR/ANLl34+Xah6Jf8CfAiCTJSRBkq7cJPq6NpKpkCKoi1zcBPIBwSA5wKNsq3UssSQvuky5qOKiKZDYeTp0f42CkdHGvjRgGeqH83hkRaO5CRS0oCPTvvjcX/vGYK8VU45CpJgSk6PRSnAoPG4wZwhxtGQQwcIdRw9yumPWIIBdfTzBULriXrAvZUuqi2I1dRMF/IIqmdlQLH78+L5eusrBw6REfoBAXoEpXRHaqgKqLoEb2gV/TmPXvv3of3OY0uednOAZqB9/ULS1Gdjg==</latexit><latexit sha1_base64="OyVw6ZLrc06cHCxa/LbkYfBkrXA=">AAACEnicbVDJSgNBFOxxjXGLevTSGAQvhhkRl4MQ8OIxglkgGUJPpydp0svY/UYIIX8heNXf8CZe/QH/wk+wk8zBJBY8KKrq8R4VJYJb8P1vb2l5ZXVtPbeR39za3tkt7O3XrE4NZVWqhTaNiFgmuGJV4CBYIzGMyEiwetS/Hfv1J2Ys1+oBBgkLJekqHnNKwElhi4ikR/ANLl34+Xah6Jf8CfAiCTJSRBkq7cJPq6NpKpkCKoi1zcBPIBwSA5wKNsq3UssSQvuky5qOKiKZDYeTp0f42CkdHGvjRgGeqH83hkRaO5CRS0oCPTvvjcX/vGYK8VU45CpJgSk6PRSnAoPG4wZwhxtGQQwcIdRw9yumPWIIBdfTzBULriXrAvZUuqi2I1dRMF/IIqmdlQLH78+L5eusrBw6REfoBAXoEpXRHaqgKqLoEb2gV/TmPXvv3of3OY0uednOAZqB9/ULS1Gdjg==</latexit>

↵ = .20<latexit sha1_base64="BR5tuePpbD+9Vmxzf0u5VSChtMk=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGaCoC6EgBuXEcwHkiG86fQkTfozdvcIIeQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VJZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VqQmtEcaWbERjKmaQ1yyynzURTEBGnjWhwO/EbT1QbpuSDHSY0FNCTLGYErJPCNvCkD/gGl8p+vlMo+iV/CrxMgowUUYZqp/DT7iqSCiot4WBMK/ATG45AW0Y4HefbqaEJkAH0aMtRCYKacDR9eoxPndLFsdJupMVT9e/GCIQxQxG5pADbN4veRPzPa6U2vgpHTCappZLMDsUpx1bhSQO4yzQllg8dAaKZ+xWTPmgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdTLpcDx+4ti5TorK4eO0Qk6QwG6RBV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CRMGdig==</latexit><latexit sha1_base64="BR5tuePpbD+9Vmxzf0u5VSChtMk=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGaCoC6EgBuXEcwHkiG86fQkTfozdvcIIeQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VJZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VqQmtEcaWbERjKmaQ1yyynzURTEBGnjWhwO/EbT1QbpuSDHSY0FNCTLGYErJPCNvCkD/gGl8p+vlMo+iV/CrxMgowUUYZqp/DT7iqSCiot4WBMK/ATG45AW0Y4HefbqaEJkAH0aMtRCYKacDR9eoxPndLFsdJupMVT9e/GCIQxQxG5pADbN4veRPzPa6U2vgpHTCappZLMDsUpx1bhSQO4yzQllg8dAaKZ+xWTPmgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdTLpcDx+4ti5TorK4eO0Qk6QwG6RBV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CRMGdig==</latexit><latexit sha1_base64="BR5tuePpbD+9Vmxzf0u5VSChtMk=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGaCoC6EgBuXEcwHkiG86fQkTfozdvcIIeQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VJZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VqQmtEcaWbERjKmaQ1yyynzURTEBGnjWhwO/EbT1QbpuSDHSY0FNCTLGYErJPCNvCkD/gGl8p+vlMo+iV/CrxMgowUUYZqp/DT7iqSCiot4WBMK/ATG45AW0Y4HefbqaEJkAH0aMtRCYKacDR9eoxPndLFsdJupMVT9e/GCIQxQxG5pADbN4veRPzPa6U2vgpHTCappZLMDsUpx1bhSQO4yzQllg8dAaKZ+xWTPmgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdTLpcDx+4ti5TorK4eO0Qk6QwG6RBV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CRMGdig==</latexit><latexit sha1_base64="BR5tuePpbD+9Vmxzf0u5VSChtMk=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGaCoC6EgBuXEcwHkiG86fQkTfozdvcIIeQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VJZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VqQmtEcaWbERjKmaQ1yyynzURTEBGnjWhwO/EbT1QbpuSDHSY0FNCTLGYErJPCNvCkD/gGl8p+vlMo+iV/CrxMgowUUYZqp/DT7iqSCiot4WBMK/ATG45AW0Y4HefbqaEJkAH0aMtRCYKacDR9eoxPndLFsdJupMVT9e/GCIQxQxG5pADbN4veRPzPa6U2vgpHTCappZLMDsUpx1bhSQO4yzQllg8dAaKZ+xWTPmgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdTLpcDx+4ti5TorK4eO0Qk6QwG6RBV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CRMGdig==</latexit>

↵ = .07<latexit sha1_base64="DYKYzhshkKmsvR26FOriwTDi6S8=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGZEiC6EgBuXEcwHkiG86XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbERjKmaQ1yyynzVhTEBGnjWh4O/EbT1QbpuSDHcU0FNCXrMcIWCeFbeDxAPANLvnlfKdQ9Ev+FHiZBBkpogzVTuGn3VUkEVRawsGYVuDHNkxBW0Y4HefbiaExkCH0actRCYKaMJ0+PcanTunintJupMVT9e9GCsKYkYhcUoAdmEVvIv7ntRLbuwpTJuPEUklmh3oJx1bhSQO4yzQllo8cAaKZ+xWTAWgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdQvSoHj95fFynVWVg4doxN0hgJURhV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CTO6djw==</latexit><latexit sha1_base64="DYKYzhshkKmsvR26FOriwTDi6S8=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGZEiC6EgBuXEcwHkiG86XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbERjKmaQ1yyynzVhTEBGnjWh4O/EbT1QbpuSDHcU0FNCXrMcIWCeFbeDxAPANLvnlfKdQ9Ev+FHiZBBkpogzVTuGn3VUkEVRawsGYVuDHNkxBW0Y4HefbiaExkCH0actRCYKaMJ0+PcanTunintJupMVT9e9GCsKYkYhcUoAdmEVvIv7ntRLbuwpTJuPEUklmh3oJx1bhSQO4yzQllo8cAaKZ+xWTAWgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdQvSoHj95fFynVWVg4doxN0hgJURhV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CTO6djw==</latexit><latexit sha1_base64="DYKYzhshkKmsvR26FOriwTDi6S8=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGZEiC6EgBuXEcwHkiG86XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbERjKmaQ1yyynzVhTEBGnjWh4O/EbT1QbpuSDHcU0FNCXrMcIWCeFbeDxAPANLvnlfKdQ9Ev+FHiZBBkpogzVTuGn3VUkEVRawsGYVuDHNkxBW0Y4HefbiaExkCH0actRCYKaMJ0+PcanTunintJupMVT9e9GCsKYkYhcUoAdmEVvIv7ntRLbuwpTJuPEUklmh3oJx1bhSQO4yzQllo8cAaKZ+xWTAWgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdQvSoHj95fFynVWVg4doxN0hgJURhV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CTO6djw==</latexit><latexit sha1_base64="DYKYzhshkKmsvR26FOriwTDi6S8=">AAACEnicbVBLSgNBFOzxG+Mv6tJNYxDcGGZEiC6EgBuXEcwHkiG86XSSJv0Zu3uEMOQWglu9hjtx6wW8hUewk8zCJBY8KKrq8R4VxZwZ6/vf3srq2vrGZm4rv72zu7dfODisG5VoQmtEcaWbERjKmaQ1yyynzVhTEBGnjWh4O/EbT1QbpuSDHcU0FNCXrMcIWCeFbeDxAPANLvnlfKdQ9Ev+FHiZBBkpogzVTuGn3VUkEVRawsGYVuDHNkxBW0Y4HefbiaExkCH0actRCYKaMJ0+PcanTunintJupMVT9e9GCsKYkYhcUoAdmEVvIv7ntRLbuwpTJuPEUklmh3oJx1bhSQO4yzQllo8cAaKZ+xWTAWgg1vU0d8VY15JxAXMuXFSZsasoWCxkmdQvSoHj95fFynVWVg4doxN0hgJURhV0h6qohgh6RC/oFb15z9679+F9zqIrXrZzhObgff0CTO6djw==</latexit>

ERM<latexit sha1_base64="2/IITHhdqgwaSXglzq1DX7A/NAA=">AAACD3icbVBLSgNBFOzxG+Mv6tLNYBDcGGZE0JUERHAjRDEfTIbQ0+lJmvRn6H4jhiGHENzqNdyJW4/gLTyCPUkWJrGgoaiqx3tdYcyZAc/7dhYWl5ZXVnNr+fWNza3tws5uzahEE1oliivdCLGhnElaBQacNmJNsQg5rYf9y8yvP1JtmJL3MIhpIHBXsogRDFZ6aIlQPaVXdzfDdqHolbwR3HniT0gRTVBpF35aHUUSQSUQjo1p+l4MQYo1MMLpMN9KDI0x6eMubVoqsaAmSEcXD91Dq3TcSGn7JLgj9e9EioUxAxHapMDQM7NeJv7nNROIzoOUyTgBKsl4UZRwF5Sbfd/tME0J8IElmGhmb3VJD2tMwJY0tcWArcjYgDkWNqpMVpE/W8g8qZ2UfMtvT4vli0lZObSPDtAR8tEZKqNrVEFVRJBEL+gVvTnPzrvz4XyOowvOZGYPTcH5+gWN4Z3m</latexit><latexit sha1_base64="2/IITHhdqgwaSXglzq1DX7A/NAA=">AAACD3icbVBLSgNBFOzxG+Mv6tLNYBDcGGZE0JUERHAjRDEfTIbQ0+lJmvRn6H4jhiGHENzqNdyJW4/gLTyCPUkWJrGgoaiqx3tdYcyZAc/7dhYWl5ZXVnNr+fWNza3tws5uzahEE1oliivdCLGhnElaBQacNmJNsQg5rYf9y8yvP1JtmJL3MIhpIHBXsogRDFZ6aIlQPaVXdzfDdqHolbwR3HniT0gRTVBpF35aHUUSQSUQjo1p+l4MQYo1MMLpMN9KDI0x6eMubVoqsaAmSEcXD91Dq3TcSGn7JLgj9e9EioUxAxHapMDQM7NeJv7nNROIzoOUyTgBKsl4UZRwF5Sbfd/tME0J8IElmGhmb3VJD2tMwJY0tcWArcjYgDkWNqpMVpE/W8g8qZ2UfMtvT4vli0lZObSPDtAR8tEZKqNrVEFVRJBEL+gVvTnPzrvz4XyOowvOZGYPTcH5+gWN4Z3m</latexit><latexit sha1_base64="2/IITHhdqgwaSXglzq1DX7A/NAA=">AAACD3icbVBLSgNBFOzxG+Mv6tLNYBDcGGZE0JUERHAjRDEfTIbQ0+lJmvRn6H4jhiGHENzqNdyJW4/gLTyCPUkWJrGgoaiqx3tdYcyZAc/7dhYWl5ZXVnNr+fWNza3tws5uzahEE1oliivdCLGhnElaBQacNmJNsQg5rYf9y8yvP1JtmJL3MIhpIHBXsogRDFZ6aIlQPaVXdzfDdqHolbwR3HniT0gRTVBpF35aHUUSQSUQjo1p+l4MQYo1MMLpMN9KDI0x6eMubVoqsaAmSEcXD91Dq3TcSGn7JLgj9e9EioUxAxHapMDQM7NeJv7nNROIzoOUyTgBKsl4UZRwF5Sbfd/tME0J8IElmGhmb3VJD2tMwJY0tcWArcjYgDkWNqpMVpE/W8g8qZ2UfMtvT4vli0lZObSPDtAR8tEZKqNrVEFVRJBEL+gVvTnPzrvz4XyOowvOZGYPTcH5+gWN4Z3m</latexit><latexit sha1_base64="2/IITHhdqgwaSXglzq1DX7A/NAA=">AAACD3icbVBLSgNBFOzxG+Mv6tLNYBDcGGZE0JUERHAjRDEfTIbQ0+lJmvRn6H4jhiGHENzqNdyJW4/gLTyCPUkWJrGgoaiqx3tdYcyZAc/7dhYWl5ZXVnNr+fWNza3tws5uzahEE1oliivdCLGhnElaBQacNmJNsQg5rYf9y8yvP1JtmJL3MIhpIHBXsogRDFZ6aIlQPaVXdzfDdqHolbwR3HniT0gRTVBpF35aHUUSQSUQjo1p+l4MQYo1MMLpMN9KDI0x6eMubVoqsaAmSEcXD91Dq3TcSGn7JLgj9e9EioUxAxHapMDQM7NeJv7nNROIzoOUyTgBKsl4UZRwF5Sbfd/tME0J8IElmGhmb3VJD2tMwJY0tcWArcjYgDkWNqpMVpE/W8g8qZ2UfMtvT4vli0lZObSPDtAR8tEZKqNrVEFVRJBEL+gVvTnPzrvz4XyOowvOZGYPTcH5+gWN4Z3m</latexit>

Page 39: John Duchi - Stanford Universityjduchi/projects/dawn-duchi.pdf · MSR Learning to Rank [Qin & Liu ’13] Beatles John atsu. ... -From 1993-2013, 98.1% of all studies on respiratory

Summary• Min. average loss often yield poor subpopulation performance

• Risk measures guarantee uniform subpopulation performance

Outliers? Learning through time? Connections to causality?

Future works

Worst subpopulation Risk-aversion

Risk-aversedecision-making

Leverage this idea in learning!

• Control minority proportions over time [HSNL’18]

• Less conservative tail-risks [DHN’18]