![Page 1: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/1.jpg)
Fairness and bias in Machine Learning
Thierry Silbermann, Tech Lead Data Science at Nubank
QCon 2019
A quick review on tools to detect biases in machine learning model
![Page 2: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/2.jpg)
Data collection• Today’s applications collect and mine vast quantities of
personal information.
• The collection and use of such data raise two important challenges.
• First, massive data collection is perceived by many as a major threat to traditional notions of individual privacy.
• Second, the use of personal data for algorithmic decision-making can have unintended and harmful consequences, such as unfair or discriminatory treatment of users.
![Page 3: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/3.jpg)
Data collection• Today’s applications collect and mine vast quantities of
personal information.
• The collection and use of such data raise two important challenges.
• First, massive data collection is perceived by many as a major threat to traditional notions of individual privacy.
• Second, the use of personal data for algorithmic decision-making can have unintended and harmful consequences, such as unfair or discriminatory treatment of users.
![Page 4: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/4.jpg)
![Page 5: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/5.jpg)
![Page 6: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/6.jpg)
![Page 7: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/7.jpg)
• Fairness is increasingly important concern as machine learning models are used to support decision making in high-stakes applications such as:
• Mortgage lending
• Hiring
• Prison sentencing
• (Approve customers, increase credit line)
Fairness
![Page 8: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/8.jpg)
Definitions of fairness
http://fairware.cs.umass.edu/papers/Verma.pdf
![Page 9: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/9.jpg)
Definitions of fairness• It is impossible to satisfy all definitions of fairness at the
same time [Kleinberg et al., 2017]
• Although fairness research is a very active field, clarity on which bias metrics and bias mitigation strategies are best is yet to be achieved [Friedler et al., 2018]
• In addition to the multitude of fairness definitions, different bias handling algorithms address different parts of the model life-cycle, and understanding each research contribution, how, when and why to use it is challenging even for experts in algorithmic fairness.
Tutorial: 21 fairness definitions and their politics: https://www.youtube.com/watch?v=jIXIuYdnyyk
![Page 10: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/10.jpg)
Example: Prison sentencing
Tutorial: 21 fairness definitions and their politics: https://www.youtube.com/watch?v=jIXIuYdnyyk
True Negative False Positive
False Negative True Positive
Did not recidivate
Recidivate
Label low-risk
Label high-risk
![Page 11: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/11.jpg)
Example: Prison sentencing
Tutorial: 21 fairness definitions and their politics: https://www.youtube.com/watch?v=jIXIuYdnyyk
True Negative False Positive
False Negative True Positive
Did not recidivate
Recidivate
Label low-risk
Label high-risk
Decision maker: Of those I’ve labeled high-risk, how many will recidivate ?
Predictive value
![Page 12: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/12.jpg)
Example: Prison sentencing
Tutorial: 21 fairness definitions and their politics: https://www.youtube.com/watch?v=jIXIuYdnyyk
True Negative False Positive
False Negative True Positive
Did not recidivate
Recidivate
Label low-risk
Label high-risk
Decision maker: Of those I’ve labeled high-risk, how many will recidivate ?
Predictive value
Defendant: What’s the probability I’ll be incorrectly classifying high-risk ?
False positive rate
![Page 13: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/13.jpg)
Example: Prison sentencing
Tutorial: 21 fairness definitions and their politics: https://www.youtube.com/watch?v=jIXIuYdnyyk
True Negative False Positive
False Negative True Positive
Did not recidivate
Recidivate
Label low-risk
Label high-risk
Decision maker: Of those I’ve labeled high-risk, how many will recidivate ?
Predictive value
Defendant: What’s the probability I’ll be incorrectly classifying high-risk ?
False positive rate
Society [think hiring rather than criminal justice]: Is the selected set demographically balanced ?
Demography
![Page 14: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/14.jpg)
https://en.wikipedia.org/wiki/Confusion_matrix
18 scores/metrics
![Page 15: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/15.jpg)
Terminology• Favorable label: a label whose value corresponds to an outcome that provides an advantage
to the recipient.
• receiving a loan, being hired for a job, and not being arrested
• Protected attribute: attribute that partitions a population into groups that have parity in terms of benefit received
• race, gender, religion
• Protected attributes are not universal, but are application specific
• Privileged value of a protected attribute: group that has historically been at a systematic advantage
• Group fairness: the goal of groups defined by protected attributes receiving similar treatments or outcomes
• Individual fairness: the goal of similar individuals receiving similar treatments or outcomes
![Page 16: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/16.jpg)
Terminology• Bias: systematic error
• In the context of fairness, we are concerned with unwanted bias that places privileged groups at a systematic advantage and unprivileged groups at a systematic disadvantage.
• Fairness metric: a quantification of unwanted bias in training data or models.
• Bias mitigation algorithm: a procedure for reducing unwanted bias in training data or models.
![Page 17: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/17.jpg)
But wait ! • I’m not using any feature that is discriminatory for my
application !
• I’ve never used gender or even race !
![Page 18: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/18.jpg)
But wait !
https://demographics.virginia.edu/DotMap/index.html
![Page 19: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/19.jpg)
But wait !
https://demographics.virginia.edu/DotMap/index.html
Chicago Area, IL, USA
![Page 20: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/20.jpg)
Fairness metric•Confusion matrix
• TP, FP, TN, FN, TPR, FPR, TNR, FNR
• Prevalence, accuracy, PPV, FDR, FOR, NPV
• LR+, LR-, DOR, F1
![Page 21: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/21.jpg)
Fairness metric•Difference of Means
•Disparate Impact
• Statistical Parity
•Odd ratios
•Consistency
•Generalized Entropy Index
![Page 22: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/22.jpg)
Statistical parity difference• Group fairness == statistical parity difference == equal acceptance rate
== benchmarking
• A classifier satisfies this definition if subjects in both protected and unprotected groups have equal probability of being assigned to the positive predicted class.
• Example, this would imply equal probability for male and female applicants to have good predicted credit score:
• P(d = 1 | G = male) = P (d = 1 | G = female)
• The main idea behind this definition is that applicants should have an equivalent opportunity to obtain a good credit score, regardless of their gender.
![Page 23: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/23.jpg)
Disparate impactX=0 X=1
Predicted condition
FALSE A B
TRUE C D
The 80% test was originally framed by a panel of 32 professionals assembled by the State of California Fair Employment Practice Commission (FEPC) in 1971
![Page 24: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/24.jpg)
Disparate impact
The 80% rule can then be quantified as:
X=0 X=1
Predicted condition
FALSE A B
TRUE C D
![Page 25: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/25.jpg)
Aequitas approach
https://dsapp.uchicago.edu/projects/aequitas/
![Page 26: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/26.jpg)
How about some solutions?
![Page 27: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/27.jpg)
Disparate impact remover
Relabelling
Learning Fair representation
![Page 28: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/28.jpg)
Disparate impact remover
Prejudice remover regulariser
Optimised Preprocessing
Relabelling
Reject Option Classification
Learning Fair representation
Adversarial Debiasing
![Page 29: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/29.jpg)
Disparate impact remover
Prejudice remover regulariser
Additive counterfactually fair estimatorOptimised Preprocessing
Equalised Odds Post-processing
Relabelling
Reweighing
Reject Option Classification
Calibrated Equalised Odds Post-processing Learning Fair representation
Adversarial Debiasing
Meta-Algorithm for Fair Classification
![Page 30: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/30.jpg)
Tools
![Page 31: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/31.jpg)
• There are three main paths to the goal of making fair predictions:
• fair pre-processing,
• fair in-processing, and
• fair post-processing
How about fixing predictions?
![Page 32: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/32.jpg)
AIF360, https://arxiv.org/abs/1810.01943
![Page 33: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/33.jpg)
Pre-Processing• Reweighing generates weights for the training examples in each
(group, label) combination differently to ensure fairness before classification.
• Optimized preprocessing (Calmon et al., 2017) learns a probabilistic transformation that edits the features and labels in the data with group fairness, individual distortion, and data fidelity constraints and objectives.
• Learning fair representations (Zemel et al., 2013) finds a latent representation that encodes the data well but obfuscates information about protected attributes.
• Disparate impact remover (Feldman et al., 2015) edits feature values to increase group fairness while preserving rank-ordering within groups.
![Page 34: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/34.jpg)
In-Processing• Adversarial debiasing (Zhang et al., 2018) learns a
classifier to maximize prediction accuracy and simultaneously reduce an adversaries ability to determine the protected attribute from the predictions. This approach leads to a fair classifier as the predictions cannot carry any group discrimination information that the adversary can exploit.
• Prejudice remover (Kamishima et al., 2012) adds a discrimination-aware regularization term to the learning objective
![Page 35: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/35.jpg)
Post-Processing• Equalized odds postprocessing (Hardt et al., 2016) solves a
linear program to find probabilities with which to change output labels to optimize equalized odds.
• Calibrated equalized odds post-processing (Pleiss et al., 2017) optimizes over calibrated classifier score outputs to find probabilities with which to change output labels with an equalized odds objective.
• Reject option classification (Kamiran et al., 2012) gives favorable outcomes to unprivileged groups and unfavorable outcomes to privileged groups in a confidence band around the decision boundary with the highest uncertainty.
![Page 36: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/36.jpg)
ExperimentsDatasets Adult Census Income, German Credit, COMPAS
Metrics
Disparate impactStatistical parity differenceAverage odds difference
Equal opportunity difference
Classifiers Logistic Regression (LR), Random Forest Classifier (RF), Neural Network (NN)
Pre-processing Algorithms
Re-weighing (Kamiran & Calders, 2012)Optimized pre-processing (Calmon et al., 2017)Learning fair representations (Zemel et al., 2013)Disparate impact remover (Feldman et al., 2015)
In-processing Algorithms
Adversarial debasing (Zhang et al., 2018)Prejudice remover (Kamishima et al., 2012)
Post-processing Algorithms
Equalized odds post-processing (Hardt et al., 2016)Calibrated eq. odds post-processing (Pleiss et al., 2017)
Reject option classification (Kamiran et al., 2012)AIF360, https://arxiv.org/abs/1810.01943
![Page 37: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/37.jpg)
Results - Statistical Parity Difference (SPD)
SPD Fair Value is 0AIF360, https://arxiv.org/abs/1810.01943
![Page 38: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/38.jpg)
Results - Disparate Impact (DI)
DI Fair Value is 1AIF360, https://arxiv.org/abs/1810.01943
![Page 39: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/39.jpg)
Results
AIF360, https://arxiv.org/abs/1810.01943
Adult census datasetProtected attribute: race
![Page 40: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/40.jpg)
Results
AIF360, https://arxiv.org/abs/1810.01943
![Page 41: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/41.jpg)
Thank you
![Page 42: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/42.jpg)
References• Conference
• ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*) https://fatconference.org/
• IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI) http://home.earthlink.net/~dwaha/research/meetings/ijcai17-xai/
• Interpretable ML Symposium - NIPS 2017 http://interpretable.ml/
![Page 43: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/43.jpg)
References• Books
• https://fairmlbook.org/
• Course materials
• Berkeley CS 294: Fairness in machine learning
• Cornell INFO 4270: Ethics and policy in data science
• Princeton COS 597E: Fairness in machine learning
![Page 44: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/44.jpg)
References• Papers
• Fairness Definitions Explained: http://fairware.cs.umass.edu/papers/Verma.pdf
• AIF360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias https://arxiv.org/pdf/1810.01943.pdf
• Aequitas: A Bias and Fairness Audit Toolkit: https://arxiv.org/pdf/1811.05577.pdf
• FairTest: Discovering Unwarranted Associations in Data-Driven Applications: https://arxiv.org/pdf/1510.02377.pdf
![Page 45: Fairness and bias in Machine Learning - QConSP...Fairness and bias in Machine Learning Thierry Silbermann, Tech Lead Data Science at Nubank QCon 2019 A quick review on tools to detect](https://reader033.vdocuments.site/reader033/viewer/2022042802/5f3b9e5ff4f06273a4573d84/html5/thumbnails/45.jpg)
References
• Videos
• Tutorial: 21 fairness definitions and their politics https://www.youtube.com/watch?v=jIXIuYdnyyk
• AI Fairness 360 Tutorial at ACM FAT* 2019 https://www.youtube.com/watch?v=XCFDckvyC0M