“fairness cases as an accelerant and enabler for cognitive assistance adoption”

© 2017 The MITRE Corporation. All rights reserved.

ChuckHowell,[email protected]

“FairnessCases”asanAccelerantandEnablerforCognitiveAssistanceAdoption*

| 2 |


Beware Disappointed Football Fans…

| 3 |


Justice Delayed is Justice Denied

| 4 |


Solution: Cool Dispassionate AI?

| 5 |


Doh!

| 6 |


Doh!

OSTPPreparingfortheFutureofAIsectiononFairness,Safety,andGovernance:

AsAItechnologiesgainbroaderdeployment,technicalexpertsandpolicyanalystshaveraisedconcernsaboutunintendedconsequences.TheuseofAItomakeconsequentialdecisionsaboutpeople,oftenreplacingdecisionsmadebyhumanactorsandinstitutions,leadstoconcernsabouthowtoensurejustice,fairness,andaccountability.

| 7 |


Proof by repeated assertion (1 of 2)

As AI systems take on a more important role in high-stakes decision-making – from offers of credit and insurance, to hiring decisions and parole – they will begin to affect who gets offered crucial opportunities, and who is left behind. This brings questions of rights, liberties, and basic fairness to the forefront.

While some hope that AI systems will help to overcome the biases that plague human decision-making, others fear that AI systems will amplify such biases, denying opportunities to the deserving and subjecting the deprived to further disadvantage.

| 8 |


Proof by repeated assertion (2 of 2)

http://www.fatml.org

| 9 |


Assertions

§ Adoption of AI in decision support and other Cognitive Assistance roles is growing significantly, and pace of adoption continues to increase

§ Concerns expressed about risks of implicit bias and discrimination

§ Point solutions being explored§ Desired: broad systems engineering framework for calibrating

and mitigating fairness risks in AI systems§ Adopting/adapting tools and techniques from safety critical

software community is one opportunity

| 10 |


Safety Critical Software§ Software is a key part of a variety of critical

systems, requiring systematic and effective techniques for assurance that the risks of deploying are understood and acceptable

§ Safety Critical Software: Software for which compelling evidence is required that it delivers a specified set of services in a manner that satisfies specified critical properties tied to safety.

“Engineers today, like Galileo three and a half centuries ago, are not superhuman. They make mistakes in their assumptions, in their calculations, in their conclusions. That they make mistakes is forgivable; that they catch them is imperative. Thus it is the essence of modern engineering not only to be able to check one’s own work, but also to have one’s work checked and to be able to check the work of others.”-- H. Petroski in To Engineer is Human: The Role of Failure in Successful Design

| 11 |


Multiple Stakeholders for Safety

Source:AndyLacher,MITRE

| 12 |


System properties vs. specific actions

§ Assessment of the system prior to operation– Various stakeholders: Are the risks of allowing this system to be

used understood, and justified by the benefits?– Certification, acceptance tests, standards compliance, etc.

§ Assessment of the results of operation– Various stakeholders: Have the claims made prior to operation

been justified? Do the benefits justify the current understanding of risks?

– Instrumentation, mishap investigation, audit, etc.

| 13 |


Some exchanges between AI and safety communities

Butmuchmoretodo,andadaptingtoothercriticalconcernsisonlystarting

| 14 |


No silver bullet, but tools and notations matter…

÷ vs.

| 15 |


Examples of opportunities to adaptsafety tools and techniques§ Fairness case framework to organize all activity and

communicate to various stakeholders§ Hazard analysis to expose potential threats to fairness (obvious

and unexpected sources)§ Instrumentation and monitoring tools and techniques to detect

potential fairness violations in operation§ Accident and incident investigation tools and techniques to

understand underlying causes of a fairness violation§ Error handling frameworks to focus engineering attention on off-

nominal cases, where problems often lurk

| 16 |


What is an “Assurance Case”?

§ Systems under regulation or acquisition constraints – Third party certification, approval, licensing, etc.– Require a documented body of evidence that provides a

compelling case that the system satisfies certain critical properties for specific contexts (to “make the case”)

– “safety case”, “certification evidence”, “security case”…– Collectively we’ll refer to them as “assurance cases”

A documented body of evidence that provides a convincing and valid argument that a specified set of critical claims about a system’s properties are adequately justified for a given system in a given environment.

| 17 |


What does tool support look like?

| 18 |


Hazard Analysis Tools

§ Standards for regulated safety-critical sectors require specific techniques for assessing the potential hazards of proposed systems. These techniques often focus on individual component failures and their associated reliability. More recent techniques such as the Systems-Theoretic Process Analysis (STPA) view safety as a system-level property, and accidents as a control problem not a component reliability problem.

§ Other emerging techniques such as Hierarchically Performed Hazard Origin & Propagation Studies (HiP-HOPS)

§ Explicitly and deliberately exploring potential fairness violations at the start could contribute to confidence in the overall fairness case, influence system design, and reduce rework

| 19 |


Hazard analysis for fairness

§ Force early consideration of well known categories of potential problems and how they will be addressed– e.g., statistical equivalence of training data wrt operational data

(“Robustness to Distributional Change”1)§ Focus on what aviation industry calls “hazardous misleading

information” not just overt failure

1ConcreteProblemsinAISafety,https://arxiv.org/abs/1606.06565#

| 20 |


Accident Investigation Tools and Notations§ Working back from an incident or accident to root causes can be

extremely expensive and complex – Millions of $s, years of effort– Consequences of false positive and negative findings– Tools and notations have evolved to help manage the data, do

“book keeping” and structural checks, and communicate complicated findings

§ Screenshots of a few follow, but the key ideas are that they are intended to support a collaborative team working backwards from a rare event through a complex, subtle, and incomplete sea of data to root causes: investigation and diagnosis

| 21 |


NASA Multi-User Investigation Organizer

| 22 |


Why-Because Analysis Graphs

| 23 |


Conclusion

§ Painfully obvious early days of a work in progress– Please help correct errors of omission and commission, all

feedback very much appreciated§ Cognitive assistance presents opportunities for great social

good, but concerns over fairness present possible impediment§ Integrated collection of tools and techniques from safety critical

software community is worth assessing – What can be readily adopted? Adapted, and how? What are gaps

that require completely new tools and techniques?

| 24 |


Closing Credits

“Theyconstantlytrytoescape...bydreamingofsystemssoperfectthatnoonewillneedtobegood”T.S.Eliot,Chorusesfrom"TheRock",VI

“Becarefulhowyoufixwhatyoudon'tunderstand.”FredBrooks,TheDesignofDesign

Thank you!


| 25 |

Backups

“fairness cases as an accelerant and enabler for cognitive assistance adoption”

Technology