an ethics code for automated and connected driving€¦ · the introduction of more highly...

Prof. Dr. Christoph Lütge

Director, TUM Institute for Ethics in Artificial Intelligence

Technical University of Munich

An Ethics Code for Automated and Connected Driving

Peter Löscher Chair of Business Ethics and Global GovernanceTUM School of GovernanceTechnical University Munich

2

Autonomous Driving: Enthusiasm and Mistrust Related to Age

39%36%

33%

24%

14%

54%

63%60%

72%

80%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

14-29 Jahre 30-39 Jahre 40-49 Jahre 50-59 Jahre 60+ Jahre

Begeisertung Misstrauen

Source: Bertelsmann Stiftung (2017), S. 3.


3

Autonomous Driving: Reasons for Skepticism in Germany

55%

56%

62%

74%

83%

84%

Verlust der Privatsphäre

Datendiebstahl

Fehlender Fahrspaß

Fremdkontrolle des Fahrzeugs durch Hackerangriffe

Verlust der eignenen Kontrolle über das Auto

Angst vor Unfällen

Source: Bertelsmann Stiftung (2017), S. 4.


Accidents involving autonomous cars:

4

Uber Volvo XC90

was hit.

▪Police: Autonomous

technology was not

responsible

▪No injuries

März 2017:

Fatal accident with

Tesla X

▪Autopilot was

activated

▪Driver ignored

several warnings

▪1 dead

23. März 2018:

Uber AV hitting

woman

▪Car driving

autonomously

▪Sensor technology

was deactivated on

purpose (false

positives)

▪1 dead

20. März 2018:

Tesla S colliding with

police car

▪Autopilot activated

▪Tesla: even with

autopilot activated,

drivers still need to

keep control of the

car

▪1 injured

29. Mai 2018:

Sources: manager magazin (2017); auto motor sport (2018); Welt (2018); tagesschau (2018).


Autonomous Driving

• Autonomous cars:

• Major problem of cars: High fatality rates

• Car accidents caused by human error by ca. 90-95%

• Autonomous cars as solution to reduce car accidents

• In Germany, this would mean ultimately saving around 3,000 lives each year – which clearly is

an ethical goal in itself.

But: Rules needed!

Prof. Dr. Christoph Lütge 5

“The introduction of more highly automated driving systems,

especially with the option of automated collision prevention, may be

socially and ethically mandated if it can unlock existing potential for

damage limitation.” (German Ethics Code for Automated and Connected Driving, 2017)


• Some problems cannot be addressed by manufacturers alone

• In particular: questions of liability:

• Who is to be held accountable in case of a crash?

• Which cars are to be licensed?

• These questions require a legal foundation in order to make

costs and benefits calculable for manufacturers

• If well-designed, such guidelines will in turn not restrain,

but accelerate the development of new technologies

Prof. Dr. Christoph Lütge | The New German Ethics Code for Autonomous Cars

The Need for a Code (for Manufacturers)

6


• More fundamentally, the introduction of autonomous cars

poses larger questions of technology management:

• How much dependence on technologically complex systems

(e.g. AI and machine learning) are we willing to accept in order

to achieve, in return, more safety, mobility and convenience?

• What precautions need to be taken to ensure controllability,

transparency and data autonomy?

• How can technological development be

advanced without blurring the contours of

a human society that places the freedom

and integrity of individuals and social

respect at the heart of its legal regime?


The Need for a Code (for Society at large)

7

Source: bpb.de


• Before 2017, a number of local detail regulations on

AVs had already been passed

• However no broader ethics code for AVs

• Against this background, the “Ethics Commission

on Automated and Connected Driving”

was appointed by the German Federal

Minister of Transport and Digital

Infrastructure in September 2016


The German Ethics Code for Automatedand Connected Driving – Procedure

8

Source: mercedes-benz.com


• The committee consisted

of 14 experts from a wide

range of disciplines:

• philosophy

• law

• the automotive industry

• consumer organisation

• In addition, external experts from other disciplines were consulted

• Head was Udo di Fabio, former Federal Constitutional Court judge


Procedure

9

Source: bmvi.de


• In June 2017, the Commission

presented its final report

• The core of the report are 20

“Ethical rules for automated

and connected vehicular traffic”:

1. The primary purpose of partly and fully automated transport

systems is to improve safety for all road users. Another purpose

is to increase mobility opportunities and to make further benefits

possible. Technological development obeys the principle of

personal autonomy, which means that individuals enjoy

freedom of action for which they themselves are responsible.


Positions: The Code

10

Source: miscw.com


2. The protection of individuals takes

precedence over all other utilitarian

considerations.

• The objective is to reduce the level of

harm until it is completely prevented

• The licensing of automated systems is

not justifiable unless it promises to

produce at least a diminution in harm

compared with human driving,

i.e. a positive balance of risks


Positions

11

Source: rci-online.org


3. The public sector is responsible for guaranteeing

the safety of the automated and connected systems

introduced and licensed in the public street environment.

• Driving systems need official licensing and monitoring

• The guiding principle is the avoidance of accidents, although

technologically unavoidable residual risks do not militate against

the introduction of

automated driving if

the balance of risks

is very positive


Positions

12

Source: imd.org; creators-rev.com


4. The personal responsibility of individuals for taking decisions is

an expression of a society centred on individual human beings.

• The purpose of all governmental and political regulatory

decisions is to promote the free development and

the protection of individuals

• Technology is to be regulated so that a balance is achieved

between maximum personal freedom of choice and the freedom

and safety of others


Positions

13


5. Automated and connected technology should prevent accidents

wherever this is practically possible.

• Based on the state of the art, the technology must be designed in

such a way that critical situations do not arise in the first place.

• the entire spectrum of technological options should be used and

constantly evolved:

• limiting the scope of application to controllable traffic environments

• preventing hazards by means of “intelligent” road infrastructure

• vehicle sensors and braking performance

• signals for persons at risk


Positions

14


6. The introduction of more highly automated driving systems,

especially with the option of automated collision prevention,

may be socially and ethically mandated if it can unlock new

potentials for damage limitation.

In contrast, mandatory fully automated transport systems

or practical inescapabilty would be ethically questionable

if it entailed submission to technological imperatives.


Positions

15

Source: scienceabc.com


7. In hazardous situations that prove to be unavoidable, the

protection of human life enjoys top priority in a balancing of

legally protected interests.

• Thus, within the constraints of what is

technologically feasible, the systems

must be programmed to accept damage

to animals or property in a conflict

if this means that personal injury

can be prevented.


Positions

16

Source: gearheads.org


8. Genuine dilemmatic decisions, such as a decision between one

human life and another, depend on the actual specific situation

and the “unpredictable” behaviour by parties affected.

They can thus not be clearly standardized, nor can they be

programmed such that they are ethically unquestionable.

• Technological systems must be

designed to avoid accidents

• they can’t replace or anticipate

the decision of a responsible

driver with the moral capacity

to make correct judgements


Positions

17

Source: autocar.co-uk


• A human driver would be acting unlawfully if he killed a person

in an emergency to save the lives of one or more other persons,

but he would not necessarily be acting culpably

• Such retrospective legal judgements cannot be transformed into

general ex ante appraisals or corresponding programming

• Rather, an independent public

sector agency should monitor

and process lessons learned

(e.g. a “Federal Bureau for the

Investigation of Accidents

Involving Automated

Transport Systems”)


Positions

18

Source: gearheads.org


9. In the event of unavoidable accident situations, any distinction

based on personal features is strictly prohibited.

• E.g. based on age, gender, physical or mental constitution

• It is also prohibited to offset victims against one another

• General programming to reduce the number

of personal injuries may be justifiable

• Those parties involved in the

generation of mobility risks

must not sacrifice non-

involved parties


Positions

19

Source: wixstatic.com


10. For automated and connected driving systems,

the accountability shifts from the motorist to:

• manufacturers and operators of technological systems

• bodies responsible for taking decisions on

infrastructure, policy and legal decisions

11. Liability for damage caused by automated driving systems is

governed by the same principles as in other product liability.

• Manufacturers and operators are thus obliged to continuously

optimize their systems and also to observe systems they have

already delivered and to improve them where this is

technologically possible and reasonable.


Positions

20

Source: eqgroup.com


12. The public is entitled to be informed about new technologies

and their deployment in a sufficiently differentiated manner.

Guidance for the deployment and

programming of automated vehicles

should be derived in a form that is

• as transparent as possible

• communicated in public

• reviewed by an independent body


Positions

21


13. The complete connectivity and central control of all

motor vehicles is ethically questionable if a total

surveillance of road users and manipulation

of vehicle control cannot safely be ruled out.

14. Automated driving is justifiable only if attacks,

in particular manipulation of the IT system or

innate system weaknesses, do not shatter

people’s confidence in road transport.

15. Owners and users decide if their vehicle data

that are generated are to be forwarded and used.


Positions

22Source: auone.jp; sophos.com; waxdigital.com


16. It must be possible to clearly distinguish if a driverless system is

being used or if a driver retains accountability with the option of

overruling the system.

• Human-machine interfaces must be designed so that it is clearly

apparent on which side individual control responsibilities lie

• Human-to-technology handovers (and vice versa) should be

documented and stored

• International standardization

of the handover and logging

procedures is to be sought


Positions

23


17. The software and technology in highly automated vehicles must

be designed such that the need for an abrupt handover of

control to the driver (“emergency”) is virtually obviated.

• To enable efficient, reliable and secure human-machine

communication and prevent overload, the systems must adapt

to human communicative behaviour (rather than vice versa)

18. Self-learning systems are only ethically

allowed if they generate safety gains.

• They must not be deployed unless they

meet the safety requirements for functions

relevant to vehicle control


Positions

24Source: roblox.com; ccli.com


19. In emergency situations, the vehicle must

autonomously, without human assistance,

enter into a “safe condition”.

• Harmonization, especially of the definition of

safe conditions or of handover routines, is desirable

20. The proper use of automated systems should

form part of people’s general digital education.

• The proper handling of automated driving systems

should be taught in during driving tuition and tested


Positions

25

Source: teknoaware.com; llowvelder.com


26

The Moral Machine (Nature 2018)

• Findings/conclusions: • Cross cultural ethical variations: 3 major

clusters of countries (see figures)

• „Differences corelate with modern

instituions and deep cultural traits“

• E.g. „Collectivist cultures, which

emphasize the respect that is due to

older members of the community,

show(ing) a weaker preference for

sparing younger characters.“

• Authors state that their findings can

„contribute to developing global,

socially acceptable principles for

machine ethics.”

Source of images (b) & quotes: Awad et al. (2018)


29

The Moral Machine (Nature 2018)

• Findings/conclusions: • Cross cultural ethical variations: 3 major

clusters of countries (see figures)

• „Differences corelate with modern

instituions and deep cultural traits“

• E.g. „Collectivist cultures, which

emphasize the respect that is due to

older members of the community,

show(ing) a weaker preference for

sparing younger characters.“

• Authors state that their findings can

„contribute to developing global,

socially acceptable principles for

machine ethics.”

Source of images (b) & quotes: Awad et al. (2018)


27

Findings of Moral Machine vs. Ethics Code

Rather kill one than many

Children should rather be

spared than elderly

How should we deal with illegally

crossing pedestrians? There are

different tendencies in different

cultures

Moral Machine Ethics Code

No recommendation

Any distinction based on

personal features, such as

age, should be prohibited vs

children

Is it possible to detect the

person who is responsible for

the dilemma situation?


28

Findings of Moral Machine vs. Ethics Code

Children should rather be

spared than elderly

How should we deal with illegally

crossing pedestrians? There are

different tendencies in different

cultures

Problems

Constitutional law does not

differentiate between young

and old when dealing with e.g.

value of life, murderer etc.

Is it possible to have programs

to act according to the drivers

morality, especially when there

are various differences in

cultures?


29

Critique: Moral Machine

• Programming Away Human Rights and

Responsibilities? (Kochupillai & Lütge, 2019)

• Well established categories of Human Rights:

• Absolute non-derogable rights – Right to Life

• Absolute derogable rights

• Qualified Rights – Right to data protection &

privacy

• Recital 4 of GDPR: “The right to the protection of

personal data is not an absolute right; it must be

considered in relation to its function in society and be

balanced against other fundamental rights, in

accordance with the principle of proportionality.”

• Article 20, GDPR: “The data subject shall have the

right to have the personal data transmitted directly

from one controller to another, where technically

feasible.”

(1) Kochupillai, M. & Lütge C. Programming Away Human Rights

& Responsibilities? A Response to Awad et al. (Forthcoming)


30

Critique: Moral Machine

• Needed: More experimental &

empirical research for determining

how to organize qualified rights to

balance out competing concerns – e.g.

privacy v. safety. (Kochupillai & Lütge,

2019)

• Focus of AVs research needs to move

away from resolving dilemma

situations to finding means of ensuring

they do not arise at all – e.g. by re-

organizing infrastructure, GDPR

compliant data collection for efficient

operation of V2x systems (Kochupillai &

Lütge, 2019)

(1) Kochupillai, M. & Lütge C. Programming

Away Human Rights & Responsibilities? A

Response to Awad et al. (Forthcoming)

Source: Tabarrock, A. The Google-Trolley

Problem, June 15, 2012

https://marginalrevolution.com/


• The German Ethics Code for Automated Driving does not

address / leaves open a large number of questions, because

• Technologies not advanced enough

• Policies are up to the political system - and not to an ethics

committee

• No agreement on ethical questions could be reached

• Examples:

• Overruling?

• Irreversibility?

• Power Centralisation?

• Mandatory Introduction at some point?


Outlook: Open Questions

31

32

Thank you very much for your attention!

Prof. Dr. Christoph Lütge

an ethics code for automated and connected driving€¦ · the introduction of more highly...

Documents