testing and evaluation methods for ict-based safety...

Testing and Evaluation Methods for ICT-based Safety Systems

Collaborative Project

Grant Agreement Number 215607

Deliverable D3.2

Final Testing Protocols

Confidentiality level: Public

Status: Final

Executive Summary This report is the final document summarizing the inspection and testing protocols of the

eVALUE project. It describes principles, inspection protocols and testing protocols for

performance testing of ICT-based safety systems. The inspection protocols (published earlier

in D2.2) and the testing protocols introduced in D3.1 are replaced by the ones in D3.2. The

older versions are obsolete and should be disregarded.

The inspection protocols cover the definition of the test vehicle, HMI aspects, environmental

conditions, and functional safety. The inspection protocols are used to prepare for the

physical tests as well as evaluating the performance of the vehicle.

The testing protocols address longitudinal, lateral, and stability-oriented traffic scenarios. The

longitudinal scenarios include a pedestrian crossing the road in front of the vehicle, or the

situation where a driver approaches a stationary queue of cars. Involuntarily lane departures

and cars in the blind spot during a lane change are situations covered by the lateral

scenarios. Exiting a highway, avoiding an obstacle, and braking on a partially ice-covered

road surface are examples of traffic scenarios related to stability.

Deliverable D3.2

eVALUE 2 ICT-2007-215607 eVALUE-101031-D32-V20-FINAL.doc www.evalue-project.eu

Document Name

eVALUE-101031-D32-V20-FINAL.doc

Version Chart

Version Date Comment

0.1 2010-10-08 First draft version

0.2 2010-10-15 Second draft released for cluster 3 comments

0.3 2010-10-25 Final draft for internal review

1.0 2010-11-05 Final version

1.1 2010-11-11 Minor corrections

1.2 2010-12-22 Updates after final review, addition of Annex C

2.0 2011-01-18 Fully revised version

Authors

The following participants contributed to this deliverable:

Name Company Chapters

Jan Jacobson, Henrik Eriksson, Jacques Hérard SP all

Susanna Leanderson-Olsson, Lars Nordström, Rafael Basso,

Krister Fredriksson VTEC all

Josep Maria Dalmau, Andrés Aparicio, Sebastien Baures, Oscar

Muñoz IDIADA all

Micha Lesemann, Adrian Zlocki, Jörn Lützow, Felix Fahrenkrog ika all

Fredrik Bruzelius, Håkan Andersson, Mattias Hjort VTI all

Daniel Westhoff SICK all

Mauro Vesco, Isabella Camuffo, Silvano Marenco, Vincenzo

Murdocco, CRF all

Javier Sanchez, Lucía Isasi RBTK all

Coordinator

Dipl.-Ing. Micha Lesemann

Institut für Kraftfahrzeuge – RWTH Aachen University

Steinbachstraße 7, 52074 Aachen, Germany

Phone: +49-241-8027535

Fax: +49-241-8022147

E-Mail: [email protected]

Copyright

© eVALUE Consortium 2010

mailto:[email protected]

Deliverable D3.2


Table of Contents

1 Introduction .................................................................................................................... 9

2 Performance testing..................................................................................................... 10

2.1 The task of performance testing ............................................................................. 10

2.2 Scenarios .............................................................................................................. 11

2.3 ICT-Based safety functions .................................................................................... 12

3 Physical testing ............................................................................................................ 14

3.1 The approach for physical testing .......................................................................... 14

3.2 Testing protocols for longitudinal functions ............................................................ 14

3.3 Testing protocols for lateral functions..................................................................... 15

3.4 Testing protocols for stability functions .................................................................. 15

4 Inspection .................................................................................................................... 16

4.1 Aims and goals of Inspections ............................................................................... 16

4.2 Inspection - Definition of Subject Vehicle ............................................................... 17

4.3 Inspection - Environmental Conditions ................................................................... 18

4.4 Inspection - HMI .................................................................................................... 19

4.5 Inspection - Functional Safety ................................................................................ 19

5 Safety performance indicators ..................................................................................... 21

5.1 Longitudinal functions ............................................................................................ 22

5.2 Lateral functions .................................................................................................... 23

5.3 Stability functions ................................................................................................... 25

6 The eVALUE Performance Testing .............................................................................. 30

7 Literature ..................................................................................................................... 33

Annex A Inspection protocols .................................................................................... 34

A.1 Inspection protocol - Definition of the subject vehicle ......................................... 35

A.1.1 Scope ...................................................................................................... 35

A.1.2 References .............................................................................................. 35

A.1.3 Definitions ................................................................................................ 37

A.1.4 Inspection ................................................................................................ 38

A.1.4.1 Principle .............................................................................................. 38

A.1.4.2 Information required from the manufacturer ......................................... 39

A.1.4.3 Inspection procedure ........................................................................... 39

A.1.4.4 Report ................................................................................................. 39

A.1.5 Checklist - Type of vehicle ....................................................................... 41

A.1.6 Checklist - Longitudinal functionality ........................................................ 43

A.1.7 Checklist - Lateral functionality ................................................................ 45

A.1.8 Checklist - Stability functionality ............................................................... 48

A.1.9 Checklist - External communication ......................................................... 50

A.2 Inspection protocol - Environmental conditions................................................... 52

A.2.1 Scope ...................................................................................................... 52

A.2.2 References .............................................................................................. 52

A.2.3 Definitions ................................................................................................ 52

A.2.4 Inspection ................................................................................................ 53

Deliverable D3.2


A.2.4.1 Principle .............................................................................................. 53



A.2.4.4 Report ................................................................................................. 54

A.2.5 Checklist - Environmental conditions ....................................................... 55

A.3 Inspection Protocol - Human-machine interface (HMI) ....................................... 58

A.3.1 Scope ...................................................................................................... 58

A.3.2 References .............................................................................................. 58

A.3.3 Definitions ................................................................................................ 59

A.3.4 Inspection ................................................................................................ 59

A.3.4.1 Principle .............................................................................................. 59



A.3.4.4 Report ................................................................................................. 60

A.3.5 Checklist - HMI ........................................................................................ 61

A.4 Inspection Protocol - Functional safety ............................................................... 68

A.4.1 Scope ...................................................................................................... 68

A.4.2 References .............................................................................................. 68

A.4.3 Definitions ................................................................................................ 69

A.4.4 Inspection ................................................................................................ 71

A.4.4.1 Principle .............................................................................................. 71



A.4.4.4 Report ................................................................................................. 72

A.4.5 Checklist - Functional safety .................................................................... 73

Annex B Testing Protocols ........................................................................................ 75

B.1 General conditions for eVALUE testing protocols ............................................... 75

B.1.1 Scope ...................................................................................................... 75

B.1.2 References .............................................................................................. 75

B.1.3 Definitions ................................................................................................ 76

B.1.4 Test conditions and environment ............................................................. 79

B.1.4.1 Test track conditions ............................................................................ 80

B.1.4.2 Temperature conditions ....................................................................... 80

B.1.4.3 Visibility conditions .............................................................................. 80

B.1.4.4 Wind conditions ................................................................................... 80

B.1.5 Subject vehicle preparation and conditioning ........................................... 80

B.1.5.1 Data collection systems ....................................................................... 81

B.1.5.2 Configuration of subject vehicle ........................................................... 81

B.2 Avoidance or mitigation of rear-end collision, open loop test .............................. 82

B.2.1 Scope ...................................................................................................... 82

B.2.2 References .............................................................................................. 82

B.2.3 Definitions ................................................................................................ 82

B.2.4 Test procedure - Avoidance or mitigation of rear-end collision ................. 83

B.2.4.1 Principle .............................................................................................. 83

Deliverable D3.2


B.2.4.2 Test objectives .................................................................................... 83

B.2.4.3 Drivers ................................................................................................. 84

B.2.4.4 Equipment ........................................................................................... 84

B.2.4.4.1 Target vehicle ............................................................................... 84

B.2.4.5 Testing environment ............................................................................ 84

B.2.4.6 Information required for the test ........................................................... 84

B.2.4.7 Subject vehicle preparation ................................................................. 85

B.2.4.8 Test procedure and data processing ................................................... 85

B.2.4.8.1 Tests ............................................................................................. 85

B.2.4.8.2 Test 1: Rear-end collision – passive driver .................................... 87

B.2.4.8.3 Test 2: Rear-end collision – driver brakes strongly ........................ 88

B.2.4.8.4 Test 3: Rear-end collision – driver brakes mildly ........................... 89

B.2.4.9 Uncertainty .......................................................................................... 90

B.2.4.10 Result ................................................................................................ 90

B.3 Avoidance of collision with transversally moving target, open loop test .............. 91

B.3.1 SCOPE .................................................................................................... 91

B.3.2 References .............................................................................................. 91

B.3.3 Definitions ................................................................................................ 91

B.3.4 Test procedure - Avoidance of collision with transversally moving target . 91

B.3.4.1 Principle .............................................................................................. 91

B.3.4.2 Test objectives .................................................................................... 92

B.3.4.3 Drivers ................................................................................................. 92

B.3.4.4 Equipment ........................................................................................... 92

B.3.4.4.1 Target vehicle ............................................................................... 93

B.3.4.4.2 Pedestrian target ........................................................................... 93

B.3.4.5 Testing environment ............................................................................ 93

B.3.4.6 Information required for the test ........................................................... 93

B.3.4.7 Subject vehicle preparation ................................................................. 93

B.3.4.8 Test procedure and data processing ................................................... 94

B.3.4.8.1 Tests ............................................................................................. 94

B.3.4.8.2 Test 1: Transversally moving target – passive driver ..................... 95

B.3.4.8.3 Test 2: Transversally moving target - driver brakes strongly.......... 96

B.3.4.8.4 Test 3: Transversally moving target – driver brakes mildly ............ 97

B.3.4.9 Uncertainty .......................................................................................... 98

B.3.4.10 Result ................................................................................................ 98

B.4 Lane departure, open loop test ........................................................................... 99

B.4.1 Scope ...................................................................................................... 99

B.4.2 References .............................................................................................. 99

B.4.3 Definitions ................................................................................................ 99

B.4.4 Test procedure - Avoidance of lane departure ......................................... 99

B.4.4.1 Principle .............................................................................................. 99

B.4.4.2 Test objectives .................................................................................. 100

B.4.4.3 Drivers ............................................................................................... 100

B.4.4.4 Equipment ......................................................................................... 100

Deliverable D3.2


B.4.4.5 Testing conditions and environment .................................................. 100

B.4.4.6 Information required for the test ......................................................... 100

B.4.4.7 Subject vehicle preparation and conditioning ..................................... 101

B.4.4.8 Test procedure and data processing ................................................. 101

B.4.4.8.1 Tests ........................................................................................... 101

B.4.4.8.2 Test 1: Lane departure on a straight road ................................... 103

B.4.4.8.3 Test 2: Lane departure in a curve................................................ 105

B.4.4.8.4 Test 3: Lane departure just before a curve .................................. 107

B.4.4.9 False alarms in Tests 1-3 .................................................................. 108

B.4.4.10 Uncertainty ...................................................................................... 108

B.4.4.11 Result .............................................................................................. 109

B.5 Avoidance of lane change collision on a straight road, open loop test .............. 110

B.5.1 Scope .................................................................................................... 110

B.5.2 References ............................................................................................ 110

B.5.3 Definitions .............................................................................................. 110

B.5.4 Test procedure - Avoidance of lane change collision on a straight road . 110

B.5.4.1 Principle ............................................................................................ 110


B.5.4.3 Drivers ............................................................................................... 111

B.5.4.4 Equipment ......................................................................................... 111

B.5.4.4.1 Target vehicle ............................................................................. 111

B.5.4.5 Testing conditions and environment .................................................. 112


B.5.4.7 Subject vehicle preparation ............................................................... 113


B.5.4.8.1 Tests ........................................................................................... 113

B.5.4.8.2 Test 1: Lane change collision avoidance on a straight road ........ 114

B.5.4.9 False alarms in Test 1 ....................................................................... 116

B.5.4.10 Uncertainty ...................................................................................... 116

B.5.4.11 Result .............................................................................................. 116

B.6 Emergency braking on a μ split, open loop test ................................................ 117

B.6.1 Scope .................................................................................................... 117

B.6.2 References ............................................................................................ 117

B.6.3 Definitions .............................................................................................. 117

B.6.4 Test procedure - Emergency braking on a μ split ................................... 117

B.6.4.1 Principle ............................................................................................ 117


B.6.4.3 Drivers ............................................................................................... 118

B.6.4.4 Equipment ......................................................................................... 118

B.6.4.5 Testing environment .......................................................................... 118




B.6.4.8.1 Tests ........................................................................................... 119

Deliverable D3.2


B.6.4.8.2 Test 1: -split braking .................................................................. 122

B.6.4.9 Uncertainty ........................................................................................ 122

B.6.4.10 Result .............................................................................................. 123

B.7 Emergency braking on a μ-split, closed loop test.............................................. 124

B.7.1 Scope .................................................................................................... 124

B.7.2 References ............................................................................................ 124

B.7.3 Definitions .............................................................................................. 124

B.7.4 Test procedure -Emergency braking on a μ-split .................................... 124

B.7.4.1 Principle ............................................................................................ 124


B.7.4.3 Drivers ............................................................................................... 125

B.7.4.4 Equipment ......................................................................................... 125





B.7.4.8.1 Tests ........................................................................................... 126

B.7.4.8.2 Test 1: -split braking .................................................................. 128

B.7.4.9 Uncertainty ........................................................................................ 129

B.7.4.10 Result .............................................................................................. 129

B.8 Obstacle avoidance .......................................................................................... 130

B.8.1 Scope .................................................................................................... 130

B.8.2 References ............................................................................................ 130

B.8.3 Definitions .............................................................................................. 130

B.8.4 Test procedure - Obstacle avoidance .................................................... 130

B.8.4.1 Principle ............................................................................................ 130


B.8.4.3 Drivers ............................................................................................... 131

B.8.4.4 Equipment ......................................................................................... 131





B.8.4.9 Uncertainty ........................................................................................ 133

B.8.4.10 Result .............................................................................................. 133

B.9 Highway exit, open loop test............................................................................. 134

B.9.1 Scope .................................................................................................... 134

B.9.2 References ............................................................................................ 134

B.9.3 Definitions .............................................................................................. 134

B.9.4 Test procedure - Highway exit ............................................................... 134

B.9.4.1 Principle ............................................................................................ 134


B.9.4.3 Drivers ............................................................................................... 135

B.9.4.4 Equipment ......................................................................................... 135

Deliverable D3.2





B.9.4.7.1 Characterization of the lateral dynamics...................................... 136


B.9.4.8.1 Tests ........................................................................................... 136

B.9.4.8.2 Test 1 Initial run .......................................................................... 138

B.9.4.8.3 Test 2 Successive runs ............................................................... 138

B.9.4.9 Uncertainty ........................................................................................ 138

B.9.4.10 Result .............................................................................................. 139

Annex C Resolved open issues ............................................................................... 140

Deliverable D3.2


1 Introduction

The eVALUE project develops test methods for Information and Communication

Technologies (ICT)-based safety systems in road vehicles. Previous deliverables of the

project have described the state-of-the-art, scenarios for testing, concepts of testing,

inspection protocols, a testing matrix, and testing protocols. Testing protocols for

performance testing exist for longitudinal, lateral, and stability scenarios. Inspection protocols

cover the definition of the test vehicle, Human-Machine Interface (HMI) aspects,

environmental aspects, and functional safety. This deliverable is the final document

summarizing the eVALUE Test Programme. It is based on definitions, results, and

conclusions from other eVALUE deliverables.

This chapter describes the purpose and the contents of the document. Chapter 2 gives the

basics of performance testing of ICT-based safety functions in road vehicles and summarizes

the scenarios previously selected for performance testing.

Chapter 3 describes physical testing and testing protocols.

Chapter 4 describes inspections for the definition of the test vehicle, the inspection of the

Human-Machine Interface (HMI), the environmental, and the functional safety aspects. (The

―inspection‖ was earlier in the project named ―design review‖. ―Inspection‖ has been chosen

since it is considered as the most appropriate name for this activity.)

Safety performance indicators are explained and listed in Chapter 5.

Chapter 6 summarises the eVALUE performance testing process.

Annex A contains the inspection protocols to be used.

Annex B contains the testing protocol describing the physical tests.

The inspection and testing protocols have been published in other eVALUE deliverables and

have been reviewed by external stakeholders for this final version.

Annex C contains a discussion on open issues.

Deliverable D3.2


2 Performance testing

2.1 The task of performance testing

Evaluation of the functional performance of a preventive safety system considers the

technical performance of the function as well as the overall safety effects; evaluation that the

function does what it was designed for. Technical performance testing aims at investigating

whether a safety function meets technical requirements and specifications on what the

function shall do [eVALUE D3.1].

Performance testing in terms of the overall safety effects aims at investigating that the

function fulfils its purpose. Interpretation of the testing result is complex because of the need

for considering driver’s response and quantifying the safety effects in terms of indicator

values derived from measurable data.

In passive safety system evaluation, safety effects are quantified by evaluating the decrease

of biomechanical injuries with e.g. airbags in crash tests. In evaluation of preventive safety

systems, intended to help avoiding or mitigating the effect of accidents, the desired safety

effects are connected to the ability to avoid an accident and to increase the driver’s control of

the situation. For such scenarios the driver’s behaviour plays an important role. Driver

behaviour modelling is very complex compared to biomechanical movements during short

lapses of time as in a crash situation. However the efforts that were invested in the eVALUE

project were to be focussed on predictable and repeatable tests that could fit within the

amount of work and time allocated to the project.

Performance testing of preventive safety systems is a way to demonstrate the performance

of the safety functions of a road vehicle. The result will be depending on the safety systems

installed in the vehicle and on the behaviour of the driver. The traffic scenarios and the test

cases chosen will give, at a reasonable test effort, an overview of the performance of the

safety functions provided by the vehicle under test. The test result is intended to be

communicated to the buyers of a vehicle but also gives information to the developer.

Development testing of preventive safety systems at a vehicle OEM or at a component

supplier is different from performance testing. During the development process an extensive

series of test cases in many different traffic scenarios need to be conducted. Thus the effort

required for development testing will be far greater than the effort required for performance

testing.

Performance testing has to be performed with limited efforts. This will also put requirements

on the efforts needed to install measuring and control equipment in the vehicle under test. It

may be time-consuming to connect to the internal data buses of the vehicle and to interpret

the messages recorded. In many cases it will also be impossible for the test engineer to have

access to the OEM-internal instructions describing the data bus communication of the

vehicle. The most efficient way to equip a vehicle would be to install a stand-alone measuring

system to record data such as position, speed, acceleration, yaw rate and distance to the

Deliverable D3.2


target. Additional sensors may have to be added to the subject vehicle to record impact with

the target vehicle. There must also be techniques to record when warning signals are issued.

The testing protocols which describe the test procedures and the test cases must be robust

to variations of test conditions. The same test results shall be obtained on different test

tracks, with different test drivers, different measurement equipment and sensors, different

target objects, at slight variations in the environmental conditions, and for different individual

vehicles of the same type with the same equipment.

A large amount of data will be recorded during the performance testing. This data has to be

possible to process and interpret in an efficient way. The logged raw data must be

transformed into a log file with measured data. The measured data will then be used to

calculate safety performance indicators describing the safety performance of the safety

function. Post-processing of measured data should be possible to automate and represent in

a clear format.

A test procedure shall fulfil the following requirements:

Cover all safety categories

High repeatability of the manoeuvre

Driver independent

Clear metric available for safety assessment

Accurate results

Reasonable test and evaluation effort

Neutral for different vehicle categories

As neutral as possible for different weather and track condition

Motivate OEM for new system introduction and improvement

Accident representative

Promote the use of the available safety features

2.2 Scenarios

Physical tests are based on scenarios that enable the evaluation of preventive safety

functions provided by a vehicle [eVALUE D1.2].

The scenarios define the context in which the testing procedures take place. They have been

selected because they permit to apply test procedures for the evaluation of the overall safety

functions in vehicles without addressing specific safety systems. The choice of scenarios is

directly based on road accident statistics. The testing procedures associated to each

scenario specify a sequence of manoeuvres from which information is derived.

The testing procedures associated to each scenario, include a description of the:

- conditions under which the tests are executed

- type of vehicle(s) involved in the tests

Deliverable D3.2


- safety indicators that quantify the overall safety performance of the current subject

vehicle

- need for measurement equipment

The preventive safety functions associated to the current scenarios are classified in three

clusters, namely:

- Cluster 1 for longitudinal assistance

- Cluster 2 for lateral assistance

- Cluster 3 for yaw/stability assistance

The final scenarios chosen by eVALUE are listed in Table 1.

Table 1 Scenarios chosen by eVALUE

Cluster Scenario

1 Scen-C1-1 Rear end collision on a straight road

Scen-C1-2 Rear end collision on a curved road

Scen-C1-3 Collision with a transversally moving target

2 Scen-C2-1 Lane departure on a straight road

Scen-C2-2 Lane departure in a curve

Scen-C2-3 Lane departure on a straight road just before a curve

Scen-C2-4 Lane change collision avoidance on a straight road

3 Scen-C3-1 Emergency braking on a μ-split

Scen-C3-2 Obstacle avoidance

Scen-C3-3 Highway exit

2.3 ICT-Based safety functions

Within the scope of eVALUE, the longitudinal domain cluster includes Forward Collision

Warning (FCW), Collision Mitigation by Braking (CMbB) and Adaptive Cruise Control (ACC).

All these functions are based on sensors which monitor the frontal environment of the vehicle

and warn, support the driver, or intervene in the case of a dangerous situation caused by an

oncoming obstacle or a pedestrian crossing the path of the subject vehicle. The proposed

test procedures are oriented to these frontal situations and emphasize the capabilities of the

sensors in this field.

The lateral domain includes Lane Departure Warning (LDW), Lane Keeping Assistant (LKA)

and Blind Spot Detection (BSD). These systems are based on sensors monitoring the side of

the vehicle and will warn, support, or intervene in case of a dangerous situation against side

obstacles or lane/road departures. For this cluster, the testing protocols try to ensure that the

safety functions are able to handle all situations where the driver lane keeping ability is

Deliverable D3.2


unintentionally lost, or when a vehicle is approaching from another lane during a lane

change.

The stability cluster evaluates the response of a vehicle in situations where the dynamics are

compromised. Electronic Stability Control (ESC) and Anti-lock Braking System (ABS) are the

systems included in this cluster.

Deliverable D3.2


3 Physical testing

3.1 The approach for physical testing

The general approach of eVALUE is based on real accident scenarios. This approach is

emphasized in physical testing, where there is a clear implementation of real traffic

scenarios. The purpose of this type of test is to assess the complete vehicle’s performance.

In other words, the approach is not to test one particular ICT-based safety system, but to

validate the whole vehicle’s safety performance under different scenarios, i.e. specific real

driving situations, which are relevant regarding the functions of the considered ICT-based

safety systems. [eVALUE D3.1]

It shall be possible to perform the complete set of eVALUE tests within maximum one week

of testing on a proving ground. Additional time will be needed for preparations, inspections

and instrumentation of the subject vehicle. The analysis of the measured data and the

reporting of the test will also require additional time. The cost of preparations, testing,

analysis and reporting is aimed to be comparable to the cost of NCAP passive safety testing

of a vehicle.

For every testing protocol there are one or more safety performance indicators. The safety

performance indicators are the key parameters which allow the evaluation of a safety

function in an objective way. They are the values which will enable the comparison of

different vehicles and show which are the strengths and weaknesses of each one, by

establishing a ranking among them, see Chapter 5. These safety performance indicators

should be consistent for different situations, and they shall be calculated from the measured

data collected during the tests.

3.2 Testing protocols for longitudinal functions

There are two testing protocols defined for the longitudinal domain:

- Avoidance of rear-end collision

- Avoidance of collision with a transversally moving target

Test procedures for rear-end collision evaluate the performance of the longitudinal safety

functions of a vehicle approaching another vehicle which is moving slower in the same

direction and lane, decelerating, or being stationary on a straight or curved road. Test

procedures for a vehicle approaching a transversally moving target address a passenger

vehicle, or a pedestrian.

Both testing protocols are open loop tests, i.e. the natural response and feedback from the

driver are not considered. Open loop tests enable the fulfilment of eVALUE requirement for

physical testing, ensuring high repeatability and driver independent procedures. Different test

cases are proposed to cover the scenarios identified for cluster 1, see Table 1.

Deliverable D3.2


3.3 Testing protocols for lateral functions

There are two testing protocols defined for the lateral domain:

- Avoidance of lane departure

- Avoidance of lane change collision on a straight road

The test procedure for avoidance of lane departure on a straight or curved road addresses

the safety of different types of subject vehicles (car, bus or truck). Lane change collision is

tested by simulating manoeuvres at lane change by a vehicle.

Both testing protocols are open loop tests, i.e. the natural response and feedback from the

driver is not expected. Different test cases are proposed to cover the scenarios identified for

cluster 2, see Table 1.

3.4 Testing protocols for stability functions

There are four testing protocols defined for the longitudinal domains:

- Emergency braking on µ-split, open loop

- Emergency braking on µ-split, closed loop

- Obstacle avoidance

- Fast driving into a curve

The emergency braking is performed both as open loop and closed loop. The obstacle

avoidance and fast driving into a curve are open loop tests. Different test cases are proposed

to cover the scenarios identified for cluster 3, see Table 1.

Deliverable D3.2


4 Inspection

4.1 Aims and goals of Inspections

An inspection is a systematic, comprehensive and documented analysis to determine the

capability and adequacy of the ICT-based safety functions. Most parts of the inspection are

done studying the documentation and further information provided by the manufacturer. The

remaining parts of the inspection might be done by investigating the vehicle [eVALUE D2.2].

An inspection may be performed with one out of three different objectives, see Figure 1. The

first objective of an inspection would be to familiarize with the subject vehicle. Efficient

performance testing will require the test engineer to understand the safety functions of the

vehicle. The design principles of the ICT-based safety systems should also be understood,

even if the performance testing does not explicitly address the performance of individual

systems. Such a ―get-to-know‖ inspection would include a definition of the subject vehicle, an

overview of functions provided for longitudinal and lateral assistance as well as an overview

of functions related to stability and external communication needs The second objective of an

inspection would be to prepare for the tests. The test cases of the performance testing will

partly be chosen depending on potential weaknesses of the system and on the already

performed tests by the manufacturer. Such an inspection concerns system performance in

different environmental conditions (rain, snow, ice, darkness, etc.). The performance in

different environmental conditions may be reviewed and assessed instead of thoroughly

tested during the performance testing. The manufacturer must be able to show the efforts

made during development testing of user behaviour and environmental conditions. ―Normal

conditions‖ (dry asphalt or concrete, flat road, approx 20°C) would then suffice at the

performance testing.

The third objective of an inspection will be to actually perform an evaluation reaching a pass

or a fail judgement. The inspection will be part of the safety evaluation. The interface to the

driver (human-machine interface, HMI) will be reviewed and assessed by a small number of

experts. A time-consuming HMI test with a large number of drivers cannot be managed

during performance testing. The focused HMI inspection by experts will conclude if the HMI

will ―pass‖ or ―fail‖ in the performance testing.

Deliverable D3.2


Figure 1: Objectives to perform an inspection (INSP)

An issue related to Inspections is the need of having a close contact with the vehicle

manufacturer. Some of the information required can be obtained by simply inspecting or

driving the vehicle. However, to get further information, it is necessary to contact the

manufacturer. At some points, it might not be easy to get information, because the

information could be sensitive, confidential, or because the manufacturer would not find the

necessity of sharing it.

4.2 Inspection - Definition of Subject Vehicle

It is important to clearly identify the vehicle under test (i.e. the subject vehicle), especially if

the vehicle is a prototype vehicle. All major characteristics of the vehicle influencing the

performance testing must be noted. The challenge of identifying the vehicle under test is

greater at performance testing than at passive safety testing. Even small design

modifications in the active safety systems may severely influence the performance.

A certain model of a road vehicle is normally produced in several different types. There will

be differences in engine, body, brakes etc. Many of these differences will not be important for

the performance testing. It will in many cases be possible to choose one type of vehicle for

the performance testing, and to assume that the test results are valid also for other types of

the same vehicle.

INSP

=

get to

know

the

subject

vehicle

INSP = evaluation

INSP

=

prepare

test

Physical testing

Deliverable D3.2


Evaluation of longitudinal, lateral, and stability safety functions require information about their

characteristics and limits, e.g. the kind of sensor used and limits of its operational range. If

some tests have been carried out by the manufacturer, it is helpful to receive information

about these tests to ensure that they are adapted to the system tested and to the related

standards.

The safety performance dependence on reliable external communication will also be

reviewed. External communication includes positioning, vehicle-to-vehicle (V2V), and

vehicle-to-infrastructure (V2I) communication. Transmission anomalies, such as distortion

and interference, can have a negative effect on the overall performance and it is

consequently addressed in the inspection.

The inspection protocol describes how to perform an inspection to identify the type of vehicle,

its safety functions, and possible dependence on external communication. The inspection is

necessary to characterise the vehicle under test and all other types of the vehicle model

intended to be covered by the performance test. It is also a necessary step for the test

engineers familiarising with the subject vehicle and its safety functions.

The results of the inspection cannot be a pass or fail judgement, but simply a definition of the

subject vehicle and its safety functions.

4.3 Inspection - Environmental Conditions

Development of ICT-based safety systems requires extensive testing to understand how the

vehicle is influenced by its environment. Temperature, light conditions, pollution,

precipitation, or road characteristics may influence the performance of the vehicle. Sensors,

controllers, and actuators might be negatively affected by adverse environmental conditions.

Development testing by the manufacturer will address all identified hazards.

Performance testing on the test track is often performed in normal environmental conditions.

Due to the limited time and resources allocated to performance testing, an inspection could

be used to validate the environmental conditions that the vehicle can handle.

The objective is to verify that the design fulfils the requirements on operation during different

environmental conditions. The important environmental influence factors must be identified,

and the vehicle and its ICT-based safety functions must be verified during different

environmental conditions.

The inspection carried out shall determine:

- if environmental conditions are expected to influence the performance of the design to be tested

- if the developer has specified requirements for environmental influence - if the developer has performed adequate tests of the environmental influence - if normal environmental conditions can be applied at performance testing on the test

track

Deliverable D3.2


4.4 Inspection - HMI

Development of ICT-based safety systems requires extensive testing to understand how the

vehicle and driver are influenced by the HMI design. The interaction between the vehicle and

the driver relates to the level of assistance provided by the vehicle which includes one or

more of the following: warning, support, or intervention.

Performance testing on the test track is often performed using robots or professional drivers.


be used to validate the suitability of the HMI design of the vehicle.

The HMI requirements specification and the user manual for the subject vehicle will be

analysed. Status indication, visual warnings, visual information, haptic warnings and auditory

warnings will be examined for the longitudinal, the lateral, and the stability safety functions.

Also the intervention and combinations of warnings/interventions will be assessed.

The inspection carried out shall determine:

- if the HMI design is expected to influence the performance of the design to be tested - if the developer has specified requirements for HMI design - if the developer has performed adequate tests of the HMI design - if simulation experiments of the HMI design is necessary to complement performance

testing on the test track

4.5 Inspection - Functional Safety

Introduction of additional safety functions in vehicles increases the complexity in safety

requirements. Preventive measures to avoid faulty states and module failures have to be

considered. Functional safety turns out to be one of the key issues as new functions as

ADAS (Advanced Driver Assistance System), dynamics control, and additional safety

systems emerge.

Faults in ICT-based safety functions may cause critical failures. The inspection of the

functional safety shall address the safety principles applied, and the safety measures

implemented to avoid hazardous system states or operation modes. The development of

ICT-based functions must be based on a requirement specification set-up adapted to the

hazardous situation attributed to systematic faults, component failures, or driver mistakes.


be used to verify how the vehicle handles different types of failure.

The objective of the inspection is to verify that the specified requirements are fulfilled when

the vehicle is operating in the occurrence of hazardous events. The important faults/failures

that may influence the behaviour of the vehicle must be identified, and the vehicle and its

ICT-based safety functions must be verified for different failure modes.

Deliverable D3.2


Failures in ICT-based safety functions are unlikely to occur during the limited time of

performance testing. Thus, an inspection is required to verify the vehicle behaviour at fault.

The inspection shall check if a standard for functional safety has been used. After

examination of the safety functions it shall be shown that:

- a situation analysis and hazard identification has been carried out - a classification of the hazards has been made - a safety integrity level is assigned to each safety function

The result of this inspection is to present an opinion on the safety-related aspects of the ICT-

based safety functions.

Deliverable D3.2


5 Safety performance indicators

Adequate safety performance indicators are essential to characterise the behaviour of the

tested vehicle according to the concept adopted in eVALUE. The number of selected

indicators of safety performance must be limited in order to reduce the complexity of the

assessment of ICT-based safety functions. A safety performance indicator shall reflect a real

impact on road safety and should not be confused with test conditions or measured values.

Test conditions are prerequisites for the test procedure e.g. the speed of the target vehicle.

Measured values are logged during the test e.g. the position of the subject vehicle. The

concept is to select the most important safety performance indicators where a real impact on

road safety can be expected [eVALUE D3.1].

An assessment of the most representative safety performance indicators is made to quantify

the overall safety performance of the vehicles. Many of the selected safety performance

indicators can be found in previous research projects or literature concerning automotive

safety development.

The approach for developing safety performance indicators involved three steps:

- The first step involved consultation of the available standards and results from

previous research projects and available assessment programme on traffic safety.

- The second step involves identifying the relevant scenarios that are representative for

the tests in which the vehicles safety functions can prevent accidents or at least

reduce their gravity.

- The third step involves the identification and development of appropriate safety

indicators. Statistics for traffic accidents involving human casualties and fatalities

have had a leading role in the final choice of safety indicators.

The safety performance indicators have been chosen to:

- Characterize the safety performance associated to the sequence of events that take

place in the current test.

- Provide information about the tested vehicle to the developer.

- Quantify the test results for comparison with a threshold value.

The approach applied, when executing the manoeuvres specified in the testing protocols, is

chosen between one of the two options; open loop or closed loop.

An open loop test enables the verification of the safety performance of the current vehicle,

without considering natural response and feedback from the driver. A professional driver or a

driving robot is used for triggering an actuation from the vehicle.

A closed loop test enables the evaluation of the safety performance of the current vehicle by

considering the driver response to eventual system activation.

Deliverable D3.2


5.1 Longitudinal functions

Two safety performance indicators are proposed for longitudinal functions, see Table 2.

Table 2 Safety performance indicators for longitudinal functions

Safety

performance

indicator

Definition/Motivation

collision speed Definition: Collision speed is the speed, at which the subject vehicle

collides into the target vehicle. It is measured at the moment of the

collision.

The relationship between the collision speed and the kinetic energy is

given by the following formula:

2

2

1collisionkin mvE

Motivation: The kinetic energy released at the moment of the collision

determines the severity of the impact between the subject vehicle and

the target vehicle.

time-to-collision

(TTC) at

warning

Definition: the point of time of the warning generation to the subject

vehicle driver. A warning signal is issued when some part of the target

vehicle lies within the alert zone or when the target vehicle is predicted

to be in the alert zone within a specified time delay after the warning

signal onset. An alert zone shall define the position at which the target

vehicle can provoke a warning signal.

A warning signal that does not fulfil this requirement is a False alarm.

Motivation: A large number of false warnings will cause the driver to

mistrust the system and ultimately the driver might turn the system off.

This safety performance indicator is introduced in deliverable D3.2 as a

result of gained experience during testing.

Deliverable D3.2


5.2 Lateral functions

Two safety performance indicators are proposed for lateral functions, see Table 3.

Table 3 Safety performance indicators for lateral functions

Safety

performance

indicator


time-to-line

crossing (TLC)

at warning

Definition: the point of time of the warning generation to the driver. A

warning signal is issued when some part of the subject vehicle lies

within the alert zone or when the subject vehicle is predicted to be in the

alert zone within a specified time delay after the warning signal onset.

An alert zone typically defines the position at which the issuing of a lane

departure warning is allowed. A warning signal that does not fulfil this

requirement is a False alarm.

Motivation: The objective of the warning signal is to support the driver

in his manoeuvres to avoid a lane departure.

A large number of false warnings will cause the driver to mistrust the

system and ultimately the driver might turn the system off. This safety

performance indicator is introduced in deliverable D3.2 as a result of

gained experience during testing.

Deliverable D3.2


Safety

performance

indicator


time until target

enters Collision

Risk Zone at

warning

Definition: the time to entering Collision Risk Zone is based on the

vehicles position and speed at the time of the warning. It depends also

on the geometry of the vehicles and the Collision Risk Zone.

The Collision Risk Zone may be stated by the OEM/system

manufacturer in terms of an official performance specification or chosen

according to an applicable standard such as ISO 17387.

The value of the time to entering Collision Risk Zone is obtained by

dividing the distance between the Collision Risk Zone boundary and the

leading or trailing edge of the target vehicle by the overtaking speed.

All warnings shall occur within an allowed warning zone (cf. ISO 17387),

which is an extension of the Collision Risk Zone. A warning signal that

does not fulfil this requirement is a False alarm.

Motivation: The objective of the warning signal is to support the driver

in his manoeuvres to avoid a lane change collision.

A large number of false warnings will cause the driver to mistrust the

system and ultimately the driver might turn the system off. This safety



Deliverable D3.2


5.3 Stability functions

Twelve safety performance indicators are proposed for stability functions, see Table 4.

Table 4 Safety performance indicators for stability functions

Safety

performance

indicator


mean

longitudinal

deceleration

Definition: the mean longitudinal deceleration is expressed by the

following formula

where

T0 is the time when driver starts to act on the brake pedal

T2 is the time when vehicle reaches the 10 km/h velocity drop off

Vx and ax follow the definitions in the glossary, expressed in [m/s] and [m/s2].

Ψ is the yaw rate [rad/sec]

Motivation: is the basis to evaluate the longitudinal performance i.e.

stopping distance. This safety performance indicator is introduced in

deliverable D3.2 as a result of gained experience during testing.

equivalent

deceleration

Definition: a parameter combining one performance measure for

stability and one performance measure for the ability to stop. It is able

to quantify the stability and the control done by the vehicle’s systems of

the available adherence of the different wheels in µ-split scenario.

2

0

2

0110 T

T

x

T

T

X

FED

dta

dtV

aa

(See variable definition listed in mean longitudinal deceleration)

Motivation: is used to evaluate the trade-off between stability and

braking performance. This safety performance indicator is introduced in


2

002

10

1T

T

xF dtaTT

a

Deliverable D3.2


Safety

performance

indicator


equivalent

deceleration on

different tracks

Definition:

(See variable definition listed in mean longitudinal deceleration)

ηHIGH and ηLOW are respectively ratios between the longitudinal

decelerations means on high and low adherence surfaces with respect

to gravity acceleration.

Motivation: aims to be a representative parameter of the vehicle

behaviour on µ split surfaces and it should be quite stable to the track

change. This safety performance indicator is introduced in deliverable

D3.2 as a result of gained experience during testing.

use of adherence Definition: The use of adherence (ε) is calculated as the quotient

between the theoretical stopping distance and the stopping distance

using the deceleration at μ-split.

Where

theorSD is the theoretical stopping distance calculated using the

average deceleration between high and low µ initial braking

manoeuvres.

µsplitSD is the stopping distance using the deceleration of the µ-split

braking manoeuvre.

Motivation: The maximum theoretical value ε=100% would represent

a very high use of adherence. The theoretical value ε=0% would

represent a very poor use of adherence. A high use of adherence

corresponds to a short stopping distance. This safety performance

indicator is introduced in deliverable D3.2 as a result of gained

experience during testing.

dtV

TTg

aa

T

T X

LOWHIGHLOWHIGH

F 2

002

10 1111

Deliverable D3.2


Safety

performance

indicator


stability Definition: the stability is expressed by the following formula;

high

lowSWRSWAStability

maxmax

Where

maxSWA is the maximum steering wheel angle achieved during the

braking manoeuvre

maxSWR is the maximum steering wheel rate achieved during the

braking manoeuvre

low is the maximum deceleration achieved on homogeneous braking

on low µ surface

high is the maximum deceleration achieved on homogeneous braking

on high µ surface

Motivation: Stability indicates the effort exerted by the driver to keep

the vehicle on the intended course. This safety performance indicator

is introduced in deliverable D3.2 as a result of gained experience

during testing.

yaw rate ratio at

COS+1s and

COS+1.75s

Definition: yaw rate is the rotation around the z axis of the vehicle.

Motivation: yaw rate describes the stability of the vehicle. This safety



lateral

displacement at

1.07s

Definition: the lateral displacement of the vehicle’s centre of gravity

with respect to its initial straight path during the portion of the sine with

dwell manoeuvre prior to beginning of the steering dwell.

Motivation: lateral displacement quantifies the vehicle

responsiveness. This safety performance indicator is introduced in


Deliverable D3.2


Safety

performance

indicator


driver intention

following

Definition: relates the difference between the real and the expected

yaw response.

time

SW

Atime

yaw

R

time

ay

time

Bra

ke

pre

ss.

time

SW

A

time

yaw

R

time

ay

time

Bra

ke p

ress.

Response gain = Response peak / SWA peak

delay

SWA peak

Response peak

time

SW

A

time

ya

wR

time

ay

time

sl R

EF

time

Bra

ke

pre

ss

time

ya

wR

diff

time

ay d

iff

time

vx

Response

control

(ay)

Response

control

Response

stability

expected responseactual responseexpected response

actual response

expected

expected - realDIF

0 expected yaw response

> 0 over responsive yaw motion

< 0 under responsive yaw motion

Motivation: expresses how well the movements of the vehicle follow the acting of the driver at the steering wheel. This safety performance indicator is introduced in deliverable D3.2 as a result of gained experience during testing.

1st steering

wheel torque

peak

Definition: corresponds to the first steering wheel torque peak

measured during the manoeuvre.

Motivation: a very high torque indicates a low possibility to control the

vehicle. This safety performance indicator is introduced in deliverable

D3.2 as a result of gained experience during testing.

wheel lift (WL) Definition: This represents the rollover condition: if both inner wheels

are lifted more than 5 cm from the ground and for more than 20 ms.

Motivation: The wheel lift is used to assess roll stability of the vehicle.

Deliverable D3.2


Safety

performance

indicator


relative radius

Rrel

Definition: The relative radius (Rrel) is the difference between the trajectory radius in the test run (Ri) and the trajectory radius in the initial test run (R1). Rrel = Ri – R1

Where R1 is the final radius value at the closing curve.

Motivation: A small relative radius value corresponds to a good

understeer control.

slip angle

(SlipCOG)

Definition: slip angle at the centre of gravity (SlipCOG) of the vehicle. Motivation: The slip angle (SlipCOG), is an indicator of the oversteer

during the test manoeuvres. A small slip angle value corresponds to a

good oversteer control. This safety performance indicator is introduced

in deliverable D3.2 as a result of gained experience during testing.

Deliverable D3.2


6 The eVALUE Performance Testing

Performance testing must be made with limited resources. Scenarios, test cases and the

number of trials have to be carefully selected. Each scenario will require a number of test

cases. Each test case will have to be performed several times to get statistically significant

data. Performance testing must concentrate on high output from a limited number of trials.

The development testing performed by vehicle manufacturers and system suppliers will be

far more time-consuming [eVALUE D3.1].

The complete eVALUE performance testing process will comprise inspections and physical

testing for the three clusters, see Figure 2. The performance testing is started by performing

two inspections.

Figure 2: The eVALUE performance testing process

The first inspection is to define the subject vehicle as a preparation for the coming tests. A

certain vehicle model may exist in several types with different level of assistance. The

inspection is to identify the type of vehicle as well as to analyse performance and

characteristics of ICT-based safety targeting longitudinal, lateral, and stability assistance.

Additionally, the safety performance demand on external communication is reviewed. The

inspection is necessary to specify the vehicle under test (the subject vehicle) and all other

types of the vehicle model for which the result of the performance test will be regarded valid.

eVALUE

Performance Testing

Inspection

Define the Subject Vehicle

Inspection

Environmental Conditions

Physical testing

Cluster 1

Longitudinal

Physical testing

Cluster 2

Lateral

Physical testing

Cluster 3

Stability

Inspection

HMI

Assessment

Test Report

Inspection

Functional Safety

Deliverable D3.2


The second inspection analyses how environmental factors influence the performance of a

vehicle. The inspection shall confirm if performance testing on the test track can be made at

normal environmental conditions.

Five actions are then taken to assess the performance. The test procedures for physical

testing in cluster 1, 2 and 3 can be performed in any order or even in parallel. Inspections for

HMI and functional safety can also be performed at the same time. The results of all five

activities are then used as input for the assessment of the ICT-based safety functions.

The eVALUE performance testing can also be described by showing the components of the

test programme, see Figure 3. Each testing protocol describes a test procedure which

contains test cases which shall be performed (each test case shall also be performed a

certain number of times, trials). The output from the testing protocols is the test results which

are used in the assessment protocols where the performance of the subject vehicle in the

current test is determined.

Testing

Protocols

Assessment

ProtocolsTest Reports

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 1

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 2

...

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE n

Test Results 1

Test Results 2

Test Results n

...

CLUSTER 1

defined in

WP3

defined in

WP4

obtained in the

execution of the

tests defined by

the project

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 1

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 2

...

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE n

Test Results 1

Test Results 2

Test Results n

...

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 1

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE 2

...

Test

Case 1

Test

Case 2

Test

Case 3

...

TEST

PROCEDURE n

Test Results 1

Test Results 2

Test Results n

...

each test

procedure is

developed in

WP2

CLUSTER 2 CLUSTER 3

Scenarios

compiled in

WP1

eVALUE Test Programme

Figure 3: The components of the eVALUE performance testing

Deliverable D3.2


The testing protocol guides the test engineer by specifying: how to prepare the subject

vehicle and test track for the tests, how to conduct the actual tests, and which test cases to

perform. The prioritisation of test cases has been made by studying and interpreting accident

data. Based on this study, assumptions concerning test conditions; vehicle speeds,

departure rates, target vehicles, etc have been made. For a test program, a reasonable

amount of test cases is necessary but shall be realisable with limited resources.

Consequently the prioritisation of test cases is important and the most representative test

cases should be selected.

Deliverable D3.2


7 Literature

[eVALUE D1.1] eVALUE.

Project deliverable D1.1 State of the Art and eVALUE scope

2008


Project deliverable D1.2 Concepts definition

2008


Project deliverable D2.2 Specifications list

2009


Project deliverable D3.1 Testing protocols, first version

2010

Deliverable D3.2


Annex A Inspection protocols

In this annex the four inspection protocols for the eVALUE inspections are presented. The

four inspections are:

1. Definition of the subject vehicle

2. Environmental conditions

3. Human-Machine Interface (HMI)

4. Functional safety

The inspection protocols have a similar structure. The first three sections are Scope,

References, and Definitions. The content of these chapters are rather self-explanatory:

Scope deals with the applicability of the inspection protocol, References lists useful

references, and Definitions defines term which are used throughout the inspection protocol.

Following that, there is a section which describes the inspection procedure, i.e. information

needed as input and how the result from the inspection shall be reported. All inspection

protocols contain one or several checklists which support the execution of the inspection.

Deliverable D3.2


A.1 Inspection protocol - Definition of the subject vehicle

A.1.1 Scope

It is important to clearly identify the vehicle under test (i.e. the subject vehicle), especially if

the vehicle is a prototype vehicle. All major characteristics of the vehicle influencing the

performance testing must be noted. The challenge of identifying the vehicle under test is

greater at performance testing than at testing of passive safety. Even small design

modifications in the active safety systems may severely influence the performance.

A certain model of a road vehicle is normally produced in several different types. It may exist

differences e.g. in engine, body and brakes. However some differences may not be important

for the performance testing. It will in many cases be possible to choose one type of vehicle

for the performance testing and assume that the test results are valid also for other types of

the same vehicle.

Evaluation of longitudinal, lateral, and stability safety functions require information about their

characteristics and limits, e.g. the kind of sensor used and limits of its operation range. If the

manufacturer has already done tests it is helpful to receive information about these tests to

ensure that they are adapted to the safety functions of the vehicle under test and to the

related standards.

The safety performance dependence on reliable external communication will also be

inspected. External communication includes positioning, vehicle-to-vehicle (V2V), and

vehicle-to-infrastructure (V2I) communication. Transmission anomalies, such as distortion

and interference, can have a negative effect on the overall performance and it is

consequently addressed in the inspection.

This testing protocol describes how to perform an inspection to identify the type of vehicle, its

safety functions, and possible dependence on external communication. The inspection is

necessary to specify the vehicle under test and all other types of the vehicle model intended

to be covered by the performance test. It is also a necessary step for the test engineers

familiarizing with the subject vehicle and its safety functions.

The inspection will be applicable for cars, trucks and buses. It shall be performed before any

physical testing or laboratory testing is performed. The results of the inspection cannot be a

pass or fail judgement, but simply a definition of the subject vehicle and its safety functions.

A.1.2 References

Longitudinal systems (e.g. ACC, FCW, and CM):

[15622] International standard ISO 15622:2002

Transport information and control systems – Adaptive Cruise Control Systems

Performance requirements and test procedures

International Standardisation Organisation, 2002

Deliverable D3.2



Transport information and control systems – Forward vehicle collision warning

systems –Performance requirements and test procedures


[J2399] SAE standard J2399

Adaptive Cruise Control (Acc) Operating Characteristics and User Interface

Society of Automotive Engineers, 2003-12

[FMC05b] Houser, A., Pierowicz, J., and McClellan, R.

Concept of Operations and Voluntary Operational Requirements for Forward

Collision Warning Systems (CWS) and Adaptive Cruise Control (ACC)

Systems On-Board Commercial Motor Vehicles.

FMCSA-MCRR-05-007, Federal Motor Carrier Safety Administration, 2005


Human Factors in Forward Collision Warning Systems: Operating

Characteristics and User Interface Requirements


Lateral systems (e.g. BSD, LDW, and LKA):


Intelligent transport systems – Lane departure warning systems –




Intelligent transport systems – Lane change decision aid systems (LCDAS)–



[FMC05a] Houser, A., Pierowicz, J., and Fuglewicz, D.

Concept of Operations and Voluntary Operational Requirements for Lane

Departure

Warning Systems (LDWS) On-Board Commercial Motor Vehicles.

FMCSA-MCRR-05-005, Federal Motor Carrier Safety Administration, 2005

[810757] Development of Crash Imminent Test Scenarios for Integrated Vehicle-Based

Safety Systems (IVBSS)

Deliverable D3.2


Publication DOT 810 757

U.S. Department of Transportation, National Highway Traffic Safety

Administration

April 2007

Download at

http://www.itsa.org/itsa/files/pdf/IVBSS%20Crash%20Imminent%20Test%20Scenario

%20Report%20-%20DOT%20HS%20810%20757.pdf


Road/Lane Departure Warning Systems: Information for the Human Interface


Stability systems (e.g. ABS and ESC):


Passenger cars – Stopping distance at straight-line braking with ABS – Open-

loop test method


[FMVSS126] FMVSS No. 126

Laboratory Test Procedure for Electronic Stability Control Systems

U.S. Department of Transportation

National Highway Traffic Safety Administration

April, 2007

Download at

http://www.nhtsa.dot.gov/staticfiles/DOT/NHTSA/Vehicle%20Safety/Test%20Procedur

es/Associated%20Files/TP126-00.pdf

A.1.3 Definitions

Model – The model of a vehicle is the basic model used as a trade name by the OEM (e.g.

Citroen C5 Tourer 2.0 16V)

Type – The type of a vehicle is one specific configuration of the vehicle model characterized

e.g. by body, engine, ICT based safety systems and brakes

Subject vehicle: Vehicle equipped with the system, which should be tested and which is

related to the topic of discussion

Target vehicle: The target vehicle reproduces the characteristics of the required vehicle

according to sensor’s technology ant the test protocol requirements.

Relative speed: The relative speed rv is the difference between the target vehicle’s speed vt

and the subject vehicle’s speed vs; rv = vs - vt

Clearance: Distance from target vehicle’s trailing surface to the subject vehicle’s leading

surface

http://www.itsa.org/itsa/files/pdf/IVBSS%20Crash%20Imminent%20Test%20Scenario%20Report%20-%20DOT%20HS%20810%20757.pdf

http://www.itsa.org/itsa/files/pdf/IVBSS%20Crash%20Imminent%20Test%20Scenario%20Report%20-%20DOT%20HS%20810%20757.pdf

http://www.nhtsa.dot.gov/staticfiles/DOT/NHTSA/Vehicle%20Safety/Test%20Procedures/Associated%20Files/TP126-00.pdf

http://www.nhtsa.dot.gov/staticfiles/DOT/NHTSA/Vehicle%20Safety/Test%20Procedures/Associated%20Files/TP126-00.pdf

Deliverable D3.2


Time gap: Time interval for travelling the clearance between consecutive vehicles. The time

gap is related to vehicle speed v and clearance c by t = c/v

Time to collision: Time interval before an impact between subject vehicle and target

vehicle, if the velocities of the subject and target vehicle are kept constant. The time to

collision is related to relative speed rv and clearance c by: tc = c/rv

Normal environmental conditions: Conditions normally applied at performance testing on

the test track:

- Test location shall be on a flat, dry asphalt or concrete surface. - Temperature 20 ˚C ±20˚C. - Horizontal visibility shall be greater than 1 km.

[Standard ISO 15622, clause 7.1]

Botts’ dots: Round non-reflective raised pavement markers

Rumble strips: Also known as audio tactile profiled markings are a road safety feature that

alerts drivers to potential danger by causing a tactile vibration and audible rumbling,

transmitted through the wheels into the car body.

A.1.4 Inspection

A.1.4.1 Principle

The vehicle OEM will supply information concerning the model of the vehicle, the type of the

vehicle and all other information on factors influencing the performance testing.

The vehicle OEM or safety system supplier will supply information about their ICT-based

safety functions. The information provided must contain results of previous tests and

simulations with an analysis on how these systems contribute to decrease accidents and

their influence on the driver. They will also supply an overview concerning the development

testing of the safety systems under different driving conditions.

The vehicle OEM or safety system supplier will provide their analysis with regard to the

communication influence on the operation of the ICT-based safety functions installed in the

vehicle. They will also supply information concerning the development testing of the safety

systems under poor transmission conditions

The inspection will include a study of the documentation provided, and contacts with

representatives of the manufacturer. The inspection is supported by checklists. The result will

be summarized in a written report.

The inspection only addresses the type of vehicle and its ICT-related safety functions.

Aspects regarding the preparation of the vehicle (load, fuelling, driving robots etc) are

specified in the testing protocols describing physical testing.

Deliverable D3.2


A.1.4.2 Information required from the manufacturer

The following information will be necessary for the inspection:

name, prototype status, body, engine, brakes, tyres

ICT-based safety functions o description of the system, its characteristics, limitations and expected

capabilities and to which standards it complies o description of test set-ups (targets, lane markings etc) o description of external communication channels used

A.1.4.3 Inspection procedure

Following aspects shall be inspected:

name/designation (e.g. Volvo FH16)

prototype vehicle

body (e.g. sedan, coupé, cabriolet, station wagon, bus, truck)

engine (e.g. diesel D16G: 540, 600, 700 hp)

brakes

tyres (e.g. winter tyres Michelin X-ice 205/60R15 )

ICT-based safety functions (e.g. ABS, ESC) o test documentation of the OEM o test conditions o system characteristics o related standards o accuracy and repetitiveness of the test o accuracy of sensors used during the test

external communication requirements

The inspection will study the documentation provided by the manufacturer. At least one

meeting with representatives of the manufacturer should be held. Working notes shall be

taken.

The inspection is supported by checklists (see appendix 1-5) listing specific aspects. For

each ICT-based safety function the checklist in appendix 5 shall be considered. The

inspection will end when the type of vehicle and the safety functions have been identified and

a report has been written.

A.1.4.4 Report

The result of the inspection will be:

- a clear identification of the subject vehicle. The identification shall include a list of all active safety functions included.

- a list of all other types of the vehicle model for which the performance testing results will be regarded valid.

- if the characteristics of the system complies with the related standard - if the inspection protocol used matches with the requirements of the standard - if the inspection protocol used by the OEM provides repeatable test results

Deliverable D3.2


- influence of the availability of communication systems on the performance of the subject vehicle

- definition of safe-state in the occurrence of communication failures/faults - performed test cases for communication shortcomings by the manufacturer

The result will be documented in a report based on the questions in the checklists in

appendices A.1.5 to A.1.9.

Deliverable D3.2


A.1.5 Checklist - Type of vehicle

n.a. = Not applicable question (some questions may not be applicable for all vehicles).

No Question Answer

Name/designation

A What trade name is used for the subject vehicle?

B Are there other trade names for the same vehicle?

C What is the date of production (model year) of the vehicle?

Prototype

D Is the vehicle a prototype vehicle? If so, please identify all active safety functions including electronic hardware and software versions extra carefully. (See question Q.)

Body

E Which body type is used for the subject vehicle? (sedan, station wagon, etc.)

F Are there other body types for which the test results should be considered valid?

G If the vehicle is a truck; what trailer is used at the performance testing?

Engine

H Which engine is used in the subject vehicle?

I Are there other engines for which the test results should be considered valid?

Brakes

J Which brakes are used in the subject vehicle?

K Are there other brakes for which the test results should be considered valid?

Deliverable D3.2


Suspension

L Which suspension is used in the subject vehicle?

M Is there other suspension for which the test results should be considered valid?

Tyres

N Which tyres (type and dimension) are used in the subject vehicle?

O Are there other tyres (types and dimensions) for which the test results should be considered valid?

ICT-based safety systems

P Which ICT-based safety functions are used in the subject vehicle?

Q Have all ICT-based safety systems been adequately identified (including electronic hardware and software versions)?

Supplemental equipment

R Is there other supplemental equipment at the subject vehicle which is standard when the vehicle is marketed?

Identification numbers

S Is there already a type approval number for the vehicle type? (The number shall be written down in inspection protocol.)

T Is there a vehicle identification number of the vehicle? (The number shall be written down in inspection protocol report.)

Deliverable D3.2


A.1.6 Checklist - Longitudinal functionality


No Question n.a. Yes No Comment

Capabilities of the subject vehicle

A1 Does the subject vehicle warn the driver when a target is detected?

A2 Does the subject vehicle warn the driver when a dangerous situation is detected?

A3 Does the safety function act on any component of the vehicle (brakes, brake lights, steering…)?

A4 Does the safety function advise the driver before actuating any component?

A5 Is the longitudinal safety function active in a specified speed range? (Or at all speeds?)

Was the subject vehicle tested with different target vehicles?

B1 Real target?

B1a Car?

B1b Truck?

B1c Motorbike?

B1d Bus?

B2 Dummy target?

B2a Car?

B2b Truck?

B2c Motorbike?

B2d Bus?

Does the subject vehicle detect different targets?

C1 Real vehicle?

C1a Car?

C1b Truck?

C1c Motorbike?

C1d Bus?

C2 Dummy Vehicle?

C3 Car?

C4 Truck?

C5 Motorbike?

C6 Bus?

C7 Pedestrian?

Which data was obtained from the tests?

D1 Speed?

D2 Relative speeds (longitudinal and lateral)?

Deliverable D3.2



D3 Acceleration lateral and longitudinal?

D4 Positions X,Y,Z of target and subject vehicle?

D5 Time?

D6 Brake light activation?

D7 Brake fluid pressure?

D8 Clearances (longitudinal and lateral)?

D9 Time gap?

D10 Time to collision?

D11 Off-board video?

D12 On-board video?

Was the subject vehicle tested with different road characteristics?

E1 Straight road?

E2 Curve road?

E3 Longitudinal slopes – ramps (hills)?

E4 Transversal slopes (in bends)?

E5 Guard rails?

E6 Signs?

E7 Lamp posts?

E8 Tunnel?

E9 Urban region (town)?

Was the subject vehicle tested with different scenarios?

F1 Following the target?

F2 Overtaking the target?

F3 Overtaken by the target?

F4 Target braking?

F5 Target accelerating?

F6 Lane change of the target?

F7 Lane change of the subject vehicle?

F8 Without target?

Test driver information

G1 Professional driver (e.g. test driver or development engineer)?

G2 Normal driver (i.e. a driver without special training for test driving)?

Test equipment information

H1 Is position information provided by a GPS or differential GPS equipment with suitable accuracy?

H2 Is speed information provided by a speed sensor with suitable accuracy?

H3 Is information provided at suitable data acquisition frequency?

Deliverable D3.2


A.1.7 Checklist - Lateral functionality




A1 Does the subject vehicle warn the driver when a lane departure is detected?

A2 Does the subject vehicle support the driver when a road departure is detected?

A3 Does the safety function act on the steering of the vehicle?

A4 Does the safety function act on the braking of the vehicle?

A5 Does the safety function advise the driver before actuating any component?

A6 Is the lateral safety function active in a specified speed range? (Or at all speeds?)

Was the subject vehicle tested with different target vehicles?

B1 Real vehicle?

B1a Car?

B1b Truck?

B1c Motorbike?

B1d Bus?

B2 Dummy Vehicle?

B2a Car?

B2b Truck?

B2c Motorbike?

B2d Bus?

Does the subject vehicle detect different targets?

C Real vehicle?

C1a Car?

C1b Truck?

C1c Motorbike?

C1d Bus?

C2 Dummy Vehicle?

C2a Car?

C2b Truck?

C2c Motorbike?

C2d Bus?


D1 Speed?

D2 Lateral relative speeds?

Deliverable D3.2



D3 Lateral acceleration?

D4 Positions X,Y,Z of target and subject vehicle?

D5 Time?

D6 Turn indicator actuation?

D7 Steering torque?

D8 Lateral clearances?

D9 Time gap?

D10 Time to collision?

D11 Off board video?

D12 On board video?

Was the subject vehicle tested with different types of lane marking?

E1 Circular reflectors?

E2 Rumble strips?

E3 Botts’ dots?

E4 Solid lines?

E5 Dashed lines?

E6 Dots?

E7 Yellow lane marks?

E8 White lane marks?

E9 Wide lane marks?

E10 Narrow lane marks?

Was the subject vehicle tested with different road characteristics?

F1 Straight road?

F2 Curve road?

F3 Hills?

F4 Guard rails?

F5 Signs?

F6 Lamp posts?

F7 Tunnel?

F8 Urban region (town)?

Was the subject vehicle tested with different scenarios?

G1 1) Lane departure?

G2 2) Road departure?

G3 3) Lane change with target vehicle overtaking?

Test driver information

H1 Driving robot?

H2 Professional driver (e.g. test driver or development engineer)?

H3 Normal driver (i.e. a driver without special training for test driving)?


Deliverable D3.2



I1 Is position information provided by GPS equipment with suitable accuracy?

I2 Is speed information provided by a speed sensor with suitable accuracy?

I3 Is lane mark information provided by a camera with suitable accuracy?

I4 Is lane mark information provided by a laser scanner with suitable accuracy?

I5 Is lane mark information provided by infrared sensor with suitable accuracy?

I6 Is lateral object information provided by a camera with suitable accuracy?

I7 Is information provided at suitable data acquisition frequency?

Deliverable D3.2


A.1.8 Checklist - Stability functionality




A1 Does the subject vehicle warn the driver when it intervenes?

A2 Has the safety function been proved to increase the stability of the vehicle?

A3 Has the safety function showed undesired interventions during tests?

Have the interactions of the safety function with other control systems been tested?

B1 Driver steering recommendation?

B2 Active front steering?

B3 Active rear steering?

B4 Active differential?

B5 Roll stability control?

B6 Trailer stability assist?

B7 Others (specify)?

Was the subject vehicle tested for different trucks configurations? (only for trucks)

C1 Tractor only?

C2 Tractor with semitrailer?

C3 Tractor with trailer?

Was the subject vehicle tested for different vehicle load conditions?

D1 Driver only?

D2 Full load condition?

D3 Other load conditions (specify)?

Was the subject vehicle tested on different road surface?

E1 Dry asphalt?

E2 Wet asphalt?

E3 Snow / ice?

E4 μ-split?

Was the subject vehicle tested for different dynamic manoeuvres?

F1 Emergency braking in straight line?

F2 Emergency braking in curve?

F3 Slow ramp steer?

F4 Step steer?

F5 Swept-sine steer?

F6 Single/double lane change?

F7 Sine with Dwell?

Deliverable D3.2



F8 ATI reversed steer?

F9 Other (specify)?


G1 Speed?

G2 Lateral acceleration?

G3 Longitudinal acceleration?

G4 Vehicle X,Y positions?

G5 Yaw rate?

G6 Sideslip angle?

G7 Corner pressures?

G8 Steering wheel angle?

Driver information

H1 Driving robot?

H2 Professional driver (e.g. test driver or development engineer)?

H3 Normal driver (i.e. a driver without special training for test driving)?


I1 Is system sensors information available (acquisition via CAN bus)?

I2 Is information provided at suitable data acquisition frequency?

Deliverable D3.2


A.1.9 Checklist - External communication

n.a. = Not applicable question (some questions may not be applicable for all vehicles.)


Requirements for external communication

A Is the safety performance of the subject vehicle dependent of external communication?

B Are requirements to availability and integrity of information/data specified?

Type of information/data

C Is the subject vehicle safety dependent on positioning information?

D Have the effect of different communication errors been analysed by the manufacturer/developer?

1) corruption 2) unintended repetition 3) incorrect sequence 4) loss 5) unacceptable delay 6) insertion 7) masquerade 8) addressing

E Has a safe state in the occurrence of communication failures been defined?

Deliverable D3.2


-

Vehicle-to-infrastructure communication

F Is the subject vehicle safety dependent of vehicle-to-roadside information?

G Have the effect of different communication errors been analysed by the manufacturer/developer?


H Has a safe-state in the occurrence of communication failures been defined?

Vehicle-to-Vehicle communication

I Is the subject vehicle safety dependent of vehicle-to-vehicle information?

J Have the effect of different communication errors been analysed by the manufacturer/developer?


K Has a safe-state in the occurrence of communication failures been defined?

Deliverable D3.2


A.2 Inspection protocol - Environmental conditions

A.2.1 Scope

Development of ICT based safety systems requires extensive testing to understand how the

vehicle is influenced by its environment. Temperature, light conditions, pollution, precipitation

or road characteristics may influence the performance. Sensors, controllers, and actuators

might be negatively affected by adverse environmental conditions. Development testing by

the manufacturer will address all identified hazards.

Performance testing on the test track is often performed at normal environmental conditions.

Due to the limited time and resources during performance testing, an inspection could be

used to validate which environmental conditions the vehicle can handle.

The objective is to verify that the design fulfils the requirements on operation during different

environmental conditions. The important environmental influence factors must be identified,

and the vehicle and its ICT-based safety systems must be verified during different


The inspection will be applicable for cars, trucks and buses. The results of the inspection will

be input to physical testing and lab testing in all clusters. If not already tested by the OEM,

the laboratory tests will be used to identify how environmental conditions affect vehicle

performance and in which way. These laboratory tests will establish the range (or limits) of

the subsequent physical tests.

A.2.2 References

[15622] International standard ISO 15622:2002 Transport information and control systems –

Adaptive Cruise Control Systems – Performance requirements and test procedures

[15623] International standard ISO 15623:2002 Transport information and control systems –

Forward vehicle collision warning systems – Performance requirements and test procedures

[17361] International standard ISO 17361:2007 Intelligent transport systems – Lane

departure warning systems – Performance requirements and test procedures

A.2.3 Definitions

normal environmental conditions : the conditions normally applied at performance testing

on the test track:

- test location shall be on a flat, dry asphalt or concrete surface. - temperature 20 ˚C ±20˚C. - horizontal visibility shall be greater than 1 km.

[standard ISO 15623, clause 6.2]

Deliverable D3.2


A.2.4 Inspection

A.2.4.1 Principle

The vehicle OEM or safety system supplier will supply their analysis how different

environmental conditions are expected to influence the safety function. They will also supply

information concerning the development testing of the safety systems during different


The inspection will include a study of the documentation provided, and an interview with

representatives of the manufacturer. The inspection is supported by a checklist. The

parameters to be analysed depend on the design validated. For example, lane markings

affect the performance of lateral safety functions but not stability safety functions.


Following information will be necessary for the inspection:

- information on which environmental parameters were identified to influence the performance

- information on how environmental influence was tested by the manufacturer Following information is optional for the inspection:

- test protocols for environmental stress tests - references to standards


Following environmental parameters shall be inspected:

temperature

light conditions

pollution

precipitation

road characteristics

miscellaneous parameters The inspection will study the documentation provided by the manufacturer. At least one


taken. It may also be helpful to drive the vehicle, if possible at different environmental

conditions.

The review is supported by a checklist (see annex) listing specific aspects associated to

environmental parameters. The inspection will end when conclusions have been drawn and a

report has been written.

Deliverable D3.2


A.2.4.4 Report

The result of the inspection will be four conclusions:

- If environmental conditions are expected to influence the performance of the design to be tested.

- If the developer has specified requirements for environmental influence. - if the developer has performed adequate tests of the environmental influence. - if normal environmental conditions can be applied at performance testing on the test

track. The result will be documented in a report based on the checklist in Appendix A.2.5.

Deliverable D3.2


A.2.5 Checklist - Environmental conditions

n.a. = Not applicable question (Some questions may not be applicable for all vehicles.)


Environmental requirements specification

A Will the performance of the vehicle be affected by environmental conditions?

B Are environmental condition parameters addressed in the requirements specification of the design?

Temperature

C Is the performance dependent on temperature conditions?

D Has the manufacturer tested the performance in the

appropriate temperature range?

1) 10 ± 30 ºC [17361]

2) 20 ºC ± 20 ºC [15623] [15622]

Light conditions

E Is the performance dependent on lighting conditions?

F Has the manufacturer tested the performance under lighting performance such as:

F1 Daylight?

F2 Night?

F3 Dusk or dawn?

G Has the manufacturer tested the performance under different sun orientation?

G1 Sun in the front?

G2 Sun in the back?

G3 Sun at left?

G4 Sun at right?

Pollution

H Is the performance affected by pollution?

I Has the manufacturer tested the performance in traffic conditions affected by pollution such as:

I1 Mud?

I2 Salt?

I3 Water spray?

I4 Smoke?

Precipitation

Deliverable D3.2



J Will the performance of the vehicle be affected by precipitation?

K Has the manufacturer tested the performance in different precipitation conditions such as:

K1 Rain?

K2 Snow?

K3 Hail?

K4 Fog?

L Has the manufacturer tested the performance on road/track with different friction such as:

L1 Ice?

L2 Wet road?

L3 Water puddle?

Wind conditions

M Will the performance of the vehicle be affected by wind conditions?

N Has the manufacturer tested the performance at different wind conditions (wind directions and wind speeds)?

Road characteristics

O Will the performance of the vehicle be affected by the road characteristics?

P Has the manufacturer tested the performance in the presence of parts of the road infrastructure such as:

P1 Signs? P2 Guard rails? P3 Lamp posts? P4 Tunnel? Q Has the manufacturer tested the performance on

different road surface materials such as:

Q1 Concrete? Q2 Asphalt? Q3 Gravel? Q4 Sand? R Has the manufacturer tested the performance at

different road inclinations?

S Has the manufacturer tested the performance at different road curvatures?

T Has the manufacturer tested the performance at different types of lane marking?

U Has manufacturer tested the performance at different types of crossing, such as:

U1 Y-crossing? U2 X-crossing? U3 T-crossing?

Deliverable D3.2



U4 roundabouts?

Miscellaneous

V Has the manufacturer identified other environmental conditions affecting the performance?

Deliverable D3.2


A.3 Inspection Protocol - Human-machine interface (HMI)

A.3.1 Scope

Development of ICT-based safety systems requires extensive testing in order to understand

how the vehicle and driver are influenced by the HMI design. Depending on the level, the

driver will be warned, supported, and/or the system intervenes in the vehicle behaviour.

Hence the interaction between the vehicle and the driver is paramount.

Physical testing on the test track is often performed using robots or professional drivers. Due

to the limited time and resources during performance testing, an inspection could be used to

validate the appropriateness of the HMI design of the vehicle.

The inspection will be applicable for cars, trucks and buses. The results of the inspection will

be input to physical testing and lab testing, simulation, in all clusters. If not already tested by

the OEM, simulations will be used to identify how HMI affect vehicle performance and in

which way. These simulations will serve as input to the subsequent physical tests.

A.3.2 References

[2575] ISO 2575: Road Vehicles – Symbols for Controls, Indicators and Telltales

[4040] ISO 4040: Road Vehicles – Location of Hand Controls, Indicators and telltales

in Motor Vehicles

[16352] ISO 16352: Road Vehicles – Ergonomic Aspects of In-Vehicle Presentation for

Transport Information and Control Systems: Warning Systems

[2402] SAE J2402: Road Vehicles – Symbols for Controls, Indicators and Telltales

[7000] ISO 7000: Graphical symbols for use on equipment

[17287] ISO 17287: Road vehicles – Ergonomic aspects of transport information and

control systems – Procedure for assessing suitability for use while driving

[9355] ISO 9355: Ergonomic requirements for the design of displays and control

actuators

[11428] ISO 11428: Ergonomics – Visual danger signals - General requirements,

design and testing

[15005] ISO 15005: Dialogue principles

Deliverable D3.2


[15622] ISO 15622: Transport information and control systems -- Adaptive Cruise

Control Systems -- Performance requirements and test procedures

[15623] ISO 15623: Transport information and control systems -- Forward vehicle

collision warning systems -- Performance requirements and test procedures

[17361] ISO 17361: Intelligent transport systems -- Lane departure warning systems --


[17387] ISO 17387: Intelligent transport systems -- Lane change decision aid systems

(LCDAS) -- Performance requirements and test procedures

A.3.3 Definitions

HMI: Human Machine Interface

A.3.4 Inspection

A.3.4.1 Principle

The vehicle OEM or safety system supplier will supply their analysis how HMI design is

expected to influence the safety function. They will also supply information concerning the

development testing of the HMI of the safety systems.

The inspection will include a study of the documentation provided, and an interview with

representatives of the manufacturer. The inspection is supported by a checklist. The

parameters to be analysed depend on the design validated.


Following information will be necessary for the inspection:

- information on which HMI parameters were identified to influence the performance - information on how HMI design was tested by the manufacturer. -

Following information is optional for the inspection:

- test protocols for HMI design - references to standards


For each cluster (longitudinal, lateral, and stability), the following HMI design aspects shall be

inspected:

Deliverable D3.2


warnings o visual o auditory o haptic

driver support

intervene o system/driver override o competition between systems

workload

system status indication

The inspection will study the documentation provided by the manufacturer. At least one


taken. It may also be helpful to examine and drive the vehicle.

The inspection is supported by a checklist (see annex) listing specific aspects associated to

HMI design. The inspection will end when conclusions have been drawn and a report has

been written.

A.3.4.4 Report

The result of the inspection will be four conclusions:

- If HMI design are expected to influence the performance of the design to be tested. - If the developer has specified requirements for HMI design. - If the developer has performed adequate tests of the HMI design. - If simulation experiments of the HMI design is necessary to complement performance

testing on the test track. The result will be documented in a report based on the checklist in Appendix A.3.5.

Deliverable D3.2


A.3.5 Checklist - HMI



HMI requirements specification

A Will the performance of the subject vehicle be affected by the HMI design?

B Is HMI design addressed in the requirements specification of the subject vehicle?

Manual

C Is the HMI well-explained in the manual of the subject vehicle?

Longitudinal Functionality

General

D Is the subject vehicle equipped with longitudinal safety functions?

Status Indication

E1 Are the modes of operation of the longitudinal safety functions presented to the driver? (E.g. activated - deactivated)

E2 Is there a possibility to turn off longitudinal safety functions?

E3 Is there a risk to turn the longitudinal safety functions off by mistake?

E4 Are the conditions of the longitudinal safety functions presented to the driver? (E.g. faulty – fault-free, capable/incapable)

Visual warnings

F1 Are there visual warnings related to longitudinal safety functions?

F2 Are standardized symbols used for visual warnings? (E.g. ISO 2575, SAE J2402)

F3 Is the size of the symbols adequate? (in relation to other warning symbols)

F4 Is the colour of the symbols adequate? (has a appropriate warning colour been selected)

F5 Is the luminance of the symbols adequate? (in relation to other warning symbols)

F6 Is the contrast of the symbols adequate? (in relation to other warning symbols)

F7 Is the viewing angle of the symbols adequate?

F8 Is the timing of symbol presentation adequate? (not too late or too early)

Deliverable D3.2



F9 Is the interval at which symbols are presented adequate?

F10 Is the location of the symbols adequate? (is head movement necessary)

F11 Is there an intensification of the warnings? (E.g. caution -> warning)

F12 Is the intensification performed adequately?

Visual information

G1 Is there visual information related to longitudinal safety functions? (e.g. menus)

G2 Is the choice of graphics/representational features suitable for what they represent?

G3 Are graphics/representational features grouped where possible?

G4 Is the information presented legible? (i.e. size, contrast, brightness, illumination, image stability, resolution, colour, etc.)

Auditory warnings

H1 Are there auditory warnings related to longitudinal safety functions?

H2 Is the auditory output (warning info) appropriate for the information to be conveyed?

H3 Is the volume of auditory warnings adequate?

H4 Is the volume adjustable by the driver?

H5 Is the tone of auditory warnings adequate? (in relation to other warning sounds)

H6 Is the timing of auditory warnings adequate? (not too late or too early)

H7 Is the interval at which auditory warnings are presented adequate?

H8 Is the location of auditory warnings adequate? (threat from one side gives warning from the same side)

H9 Is there an intensification of the warnings? (E.g. caution -> warning)

H10 Is the intensification performed adequately?

Haptic warnings

I1 Are there haptic warnings related to longitudinal safety functions?

I2 Is the intensity used for haptic warnings adequate? (possible to feel during normal driving conditions)

I3 Is the timing of haptic warnings adequate? (not too late or too early)

I4 Is the interval at which haptic warnings are presented adequately?

I5 Is the location of haptic warnings adequate? (threat from one side gives warning from the same side)

I6 Is there an intensification of the warnings? (E.g. caution -> warning)

Deliverable D3.2



I7 Is the intensification performed adequately?

Interventions

J1 Are there interventions related to longitudinal safety functions?

J2 Can the driver override the safety functions if needed?

J3 Can the safety functions override the driver?

Lateral Functionality

General

K Is the subject vehicle equipped with lateral safety functions?

Status Indication

L1 Are the modes of operation of the lateral safety functions presented to the driver? (E.g. activated - deactivated)

L2 Is there a possibility to turn off lateral safety functions?

L3 Is there a risk to turn the lateral safety functions off by mistake?

L4 Are the conditions of the lateral safety functions presented to the driver? (E.g. faulty – fault-free, capable/incapable)

Visual warnings

M1 Are there visual warnings related to lateral safety functions?

M2 Are standardized symbols used for visual warnings? (E.g. ISO 2575, SAE J2402)

M3 Is the size of the symbols adequate? (in relation to other warning symbols)

M4 Is the colour of the symbols adequate? (has a appropriate warning colour been selected)

M5 Is the luminance of the symbols adequate? (in relation to other warning symbols)

M6 Is the contrast of the symbols adequate? (in relation to other warning symbols)

M7 Is the viewing angle of the symbols adequate?

M8 Is the timing of symbol presentation adequate? (not too late or too early)

M9 Is the interval at which symbols are presented adequate?

M10 Is the location of the symbols adequate? (is head movement necessary)

M11 Is there an intensification of the warnings? (E.g. caution -> warning)

M12 Is the intensification performed adequately?

Deliverable D3.2



Visual information

N1 Is there visual information related to lateral safety functions? (e.g. menus)

N2 Is the choice of graphics/representational features suitable for what they represent?

N3 Are graphics/representational features grouped where possible?

N4 Is the information presented legible? (i.e. size, contrast, brightness, illumination, image stability, resolution, colour, etc.)

Auditory warnings

O1 Are there auditory warnings related to lateral safety functions?

O2 Is the auditory output (warning info) appropriate for the information to be conveyed?

O3 Is the volume of auditory warnings adequate? (possible to hear during normal driving conditions)

O4 Is the volume adjustable by the driver?

O5 Is the tone of auditory warnings adequate? (in relation to other warning sounds)

O6 Is the timing of auditory warnings adequate? (not too late or too early)

O7 Is the interval at which auditory warnings are presented adequate?

O8 Is the location of auditory warnings adequate? (threat from one side gives warning from the same side)

O9 Is there an intensification of the warnings? (E.g. caution -> warning)

O10 Is the intensification performed adequately?

Haptic warnings

P1 Are there haptic warnings related to lateral safety functions?

P2 Is the intensity used for haptic warnings adequate? (possible to feel during normal driving conditions)

P3 Is the timing of haptic warnings adequate? (not too late or too early)

P4 Is the interval at which haptic warnings are presented adequately?

P5 Is the location of haptic warnings adequate? (threat from one side gives warning from the same side)

P6 Is there an intensification of the warnings? (E.g. caution -> warning)

P7 Is the intensification performed adequately?

Interventions

Q1 Are there interventions related to lateral safety functions?

Deliverable D3.2



Q2 Can the driver override the safety functions if needed?

Q3 Can the safety functions override the driver?

Stability Functionality

General

R Is the subject vehicle equipped with stability safety functions?

Status Indication

S1 Are the modes of operation of the stability safety functions presented to the driver? (E.g. activated - deactivated)

S2 Is there a possibility to turn off stability safety functions?

S3 Is there a risk to turn the stability safety functions off by mistake?

S4 Are the conditions of the stability safety functions presented to the driver? (E.g. faulty – fault-free, capable/incapable)

Visual warnings

T1 Are there visual warnings related to stability safety functions?

T2 Are standardized symbols used for visual warnings? (E.g. ISO 2575, SAE J2402)

T3 Is the size of the symbols adequate? (in relation to other warning symbols)

T4 Is the colour of the symbols adequate? (has a appropriate warning colour been selected)

T5 Is the luminance of the symbols adequate? (in relation to other warning symbols)

T6 Is the contrast of the symbols adequate? (in relation to other warning symbols)

T7 Is the viewing angle of the symbols adequate?

T8 Is the timing of symbol presentation adequate? (not too late or too early)

T9 Is the interval at which symbols are presented adequate?

T10 Is the location of the symbols adequate? (is head movement necessary)

Visual information

U1 Is there visual information related to stability safety functions? (e.g. menus)

U2 Is the choice of graphics/representational features suitable for what they represent?

Deliverable D3.2



U3 Are graphics/representational features grouped where possible?

U4 Is the information presented legible? (i.e. size, contrast, brightness, illumination, image stability, resolution, colour, etc.)

Auditory warnings

V1 Are there auditory warnings related to stability safety functions?

V2 Is the auditory output (warning info) appropriate for the information to be conveyed?

V3 Is the volume of auditory warnings adequate? (possible to hear during normal driving conditions)

V4 Is the volume adjustable by the driver?

V5 Is the tone of auditory warnings adequate? (in relation to other warning sounds)

V6 Is the timing of auditory warnings adequate? (not too late or too early)

V7 Is the interval at which auditory warnings are presented adequate?

V8 Is there an intensification of the warnings? (E.g. caution -> warning)

V9 Is the intensification performed adequately?

Haptic warnings

W1 Are there haptic warnings related to stability safety functions?

W2 Is the intensity used for haptic warnings adequate? (possible to feel during normal driving conditions)

W3 Is the timing of haptic warnings adequate? (not too late or too early)

W4 Is the interval at which haptic warnings are presented adequately?

W5 Is the location of haptic warnings adequate? (threat from one side gives warning from the same side)

W6 Is there an intensification of the warnings? (E.g. caution -> warning)

W7 Is the intensification performed adequately?

Intervention

X1 Are there interventions related to stability safety functions?

X2 Can the driver override the safety functions if needed?

X3 Can the safety functions override the driver?

Combinations

Warnings

Y1 Could there be combination of warnings? (from two or more safety functions)

Deliverable D3.2



Y2 Are priorities of warnings handled adequately? (i.e. priority is given to the warning related to the function of highest safety relevance)

Y3 Can warnings be discriminated from each other?

Y4 Is there a risk that the workload of the driver becomes too high? (i.e. safety function(s) present(s) excessively distracting information)

Interventions

Z1 Could there be combination of interventions? (from two or more safety functions)

Z2 Are priorities of interventions handled adequately?

Deliverable D3.2


A.4 Inspection Protocol - Functional safety

A.4.1 Scope

More and more vehicle safety functions are implemented using ICT-based systems. As a

consequence, functional safety requirements, preventive actions to avoid faulty states, and

failures of those systems become key issues.

A failure in ICT-based safety functions is a potential source of hazards. The inspection of the

functional safety shall address the safety principles applied and the safety measures

implemented to avoid hazardous system states or operation modes.

The inspection addresses three aspects:

- Identification of safety functions - Independent evaluation according to the ISO 26262 or IEC 61508 standards - Hazard analysis


be used to verify how the vehicle handles different types of failure.

The objective is to verify that functional safety was considered during the development. The

important faults/failures that may influence the behaviour of the vehicle must be identified,

and the vehicle and its ICT-based safety functions must be verified for different failure

modes.

Failures in ICT-based safety functions are unlikely to occur during performance testing. Thus,

an inspection is required to verify the vehicle behaviour at fault.

The inspection will be applicable for cars, trucks and buses.

A.4.2 References

[26262] ISO/DIS 26262 Road vehicles – Functional Safety, Rev.1 – Part 1 to Part 9,

2008-02-29

[26262-1] ISO/DIS 26262-1 Road vehicles – Functional Safety – Part 1: Glossary

[26262-2] ISO/DIS 26262-2 Road vehicles – Functional Safety – Part 2: Management of

functional safety.

[26262-3] ISO/DIS 26262-3 Road vehicles – Functional Safety – Part 3: Concept phase

[26262-4] ISO/DIS 26262-4 Road vehicles – Functional Safety – Part 4: Product

development system level

Deliverable D3.2


[IEC 61508] IEC 61508:2010 Functional safety of electrical/electronic /programmable

electronic safety-related systems

A.4.3 Definitions

ASIL: the Automotive Safety Integrity Levels are determined by a systematic evaluation of

potentially hazardous operational situations. The ASIL-Level shall be determined for each

hazardous event using the estimation parameters severity (S), probability of exposure (E)

and controllability (C) in accordance with Table 4. [26262-3]

Controllability (C): The controllability shall be assigned to one of the controllability classes

C0, C1, C2 and C3 in accordance with Table 3. If a situation is regarded as simply distracting

or disturbing but as controllable in general, the class C0 may be assigned. If a hazard is

assigned to controllability class C0, no ASIL assignment is required. [26262-3]

Element: sub-units of items including system, subsystem, component. [26262-3]

E/E system: system that consists of electrical and/or electronic elements, including

programmable electronic elements, power supplies, sensors and other input devices, data

highways and other communication paths, and actuators and output devices. [26262-3]

Functional safety: absence of unacceptable risk due to hazards caused by mal-function

behaviour of E/E systems. [26262-3]

Functional safety concept: specification of the functional safety requirements, their

allocation to architectural elements and their interaction necessary to achieve the safety

goals, and information associated with these requirements.

Functional safety requirements: specification of implementation-independent safety-

related behaviour or a safety measure including its safety-related attributes. [26262-3]

NOTE 1 Safety requirement implemented by a safety-related E/E system or by a safety-related system of other

technologies in order to achieve or maintain a safe state for the item taking into account a determined hazardous

event.

NOTE 2 The functional safety requirements are specified independent of the used technology in the concept

phase of product development. They are detailed into technical safety requirements after concept phase.

Hazard classification: the hazard classification scheme comprises the determination of the

severity (S), the probability of exposure (E) and the controllability (C) associated with the

considered hazard of the item. For a given hazard, this classification will result in one or

more combinations of S, E and C classes. Each such combination represents an estimate of

a potential harm in a particular driving situation, with the severity determined by the potential

harm and the exposure determined by the situation. The controllability rates how easy or

difficult it is for the driver or other road traffic participant to avoid the considered accident type

in the considered situation. [26262-3]

Deliverable D3.2


Probability of exposure (E): The probability of exposure of the operational situations shall

be estimated. The probability of exposure shall be assigned to one of the probability classes

E1, E2, E3 and E4 in accordance with Table 2. [26262-3]

Risk: combination of the probability of occurrence of harm and the severity of that harm

[ISO/IEC Guide 51:1999, definition 3.2]

Safety: Freedom from unacceptable risk of physical injury or of damage to the health of

people, either directly or indirectly as a result of damage to property or to the environment.

Situation analysis and hazard identification: The operational situations and operating

modes in which the item is able to trigger hazards shall be described for cases when

correctly used, when incorrectly used in a foreseeable way, or in case of item failure. [26262-

3]

Severity (S): The severity of potential harm shall be estimated. The severity shall be

assigned to one of the severity classes S0, S1, S2 or S3 in accordance with Table 1. [26262-

3]

Situation analysis and hazard identification: The operational situations and operating

modes in which the item is able to trigger hazards shall be described for cases when

correctly used, when incorrectly used in a foreseeable way, or in case of item failure. [26262-

3]

Table 5 Classes of severity

Class S0 S1 S2 S3

Description No injuries

Light and moderate injuries

Severe and life-threatening injuries (survival probable)

Life-threatening injuries (survival uncertain), fatal injuries

Table 6 Classes of probability of exposure regarding operational situations

Class E1 E2 E3 E4

Description Very low probability

Low probability Medium probability

High probability

Table 7 Classes of controllability

Class C0 C1 C2 C3

Description

Controllable in general

Simply controllable

Normally controllable

Difficult to control or uncontrollable

Deliverable D3.2


C1 C2 C3

S1

E1 QM QM QM

E2 QM QM QM

E3 QM QM A

E4 QM A B

S2

E1 QM QM QM

E2 QM QM A

E3 QM A B

E4 A B C

S3

E1 QM QM A

E2 QM A B

E3 A B C

E4 B C D

Figure 1 ASIL determination

A.4.4 Inspection

A.4.4.1 Principle

The inspection is based on a study of the documents provided by the vehicle OEM or safety

system suppliers, and/or an interview with representatives of the manufacturer. The

inspection is supported by a checklist.


The documents required to carry out the inspection, shall provide the following information:

- identification of the safety functions - description on how the safety functions are integrated in the vehicle - validation according to applicable standards


The present inspection comprises two phases. The first phase addresses the identification of

the safety functions. In the second phase, the fulfilment of existing automotive standard

requirements is verified.

Questions A to D support the identification of the safety functions and their integration into

the vehicle.

Questions E and F verifies the application of relevant safety development standard. Two

standards are applicable:

- ISO 26262 and

Deliverable D3.2


- IEC 61508.

Safety functions for which an independent evaluation according to one of the above

mentioned standards has been performed, no supplementary verification is required in this

inspection.

In the absence of an independent evaluation according to a specific standard, questions G to

U permit to verify how the management process and the safety requirements have been

handled.

The different safety aspects are verified as follows in the checklist:

- Implementation of the situation analysis and hazard identification, questions G to L.

- Hazard classification, questions M to P

- ASIL determination, questions Q and R

- Fault detection capacities, questions: S and T.

- False alarms, question U.

Safety principles, hardware design and software design are not parts of this inspection.

A.4.4.4 Report

The inspection shall provide information about the examination of the safety functions and

validate that an independent evaluation based on the ISO 26262 or IEC 61508 has been

conducted:

- An independent evaluation has been carried out.

- A classification of the hazards has been made

The result of this inspection is to deliver a statement on the functional safety of the ICT-

related systems.

Deliverable D3.2


A.4.5 Checklist - Functional safety


Safety functions and independent evaluation


A Are ICT-based safety functions implemented in the vehicle?

B Is each ICT-based safety function developed according to specific international or European standard?

C Are all ICT-based safety functions identified and documented?

D Are the test results of the ICT-based function integration at system and vehicle level documented?

E Have the safety functions been subject to an independent evaluation according to the ISO 26262 development standard?

F Have the safety functions been subject to an independent evaluation according to the IEC 61508 development standard?

If independent evaluation is missing

Situation analysis and hazard identification


G Are the hazard analysis and risk assessment fully documented with regard to situation analysis and hazard identification?

H Have the potential unintended functional states that could lead to a hazard event been identified?

I Has foreseeable driver use and misuse been considered in the situation analysis and hazard identification?

J Has the impact of high speed driving been considered in the situation analysis and hazard identification?

K Has urban driving been considered in the situation analysis and hazard identification?

L Has the interaction between operational systems been considered in the situation analysis and hazard identification?

Deliverable D3.2


Hazard classification

M Is the hazard analysis and risk assessment fully documented with regard to hazard classification?

N Has the severity (S) associated with the considered hazard been determined?

O Has the probability of exposure (E) associated with the considered hazard been determined?

P Has the controllability (C) associated with the considered hazard been determined?

ASIL determination

Q Is the Hazard analysis and Risk assessment fully documented with regard to ASIL determination?

R Has the ICT-based safety function been assigned a safety integrity level (ASIL)?

Fault detection capabilities

S Has the manufacturer identified the failure modes of the system?

T Has the manufacturer identified the causes and effects of such failures?

False alarms

U Has the manufacturer tested and analysed the identified situations where a false alarm occurs?

Deliverable D3.2


Annex B Testing Protocols

B.1 General conditions for eVALUE testing protocols

B.1.1 Scope

This part addresses general conditions for eVALUE tests and includes the following sections:

References

Definitions

Test conditions & environment

Subject vehicle preparation & conditioning

B.1.2 References

[ECE 384] European agreement on main international traffic arteries, ECE/TRANS/SC.1/384,

Economic Commission for Europe Inland Transport Committee, 2008.

[NHTSA ESC] Electronic Stability Control NCAP Confirmation Test, March 2010, NHTSA.

[NHTSA FCW] Forward Collision Warning System NCAP Confirmation Test, March 2010,

NHTSA.

[NHTSA LDW] Lane Departure Warning System NCAP Confirmation Test, March 2010,

NHTSA.

[ISO 8855] Road vehicles – Vehicle dynamics and road-holding - Vocabulary (ISO

8855:1991)

[15037-1] Road vehicles – Vehicles dynamics test methods – Part 1: General conditions for

passenger cars (ISO 15037-1:2006, IDT)

[15037-2] Road vehicles — Vehicle dynamics test methods – Part 2: General conditions for

heavy vehicles and busses (ISO 15037-2:2007).

[15622] Transport information and control systems – Adaptive cruise control systems -

Performance requirements and test procedures (ISO 15622:2002)

[15623] Transport information and control systems – Forward vehicle collision warning

systems – Performance requirements and test procedures (ISO 15623:2002)

[17361] Intelligent transport systems – Lane departure warning system (LDWS) –

Performance requirements and test procedures (ISO 17361:2007, IDT)

[17387] Intelligent transport systems – Lane change decision aid systems – Performance

requirements and test procedures (ISO 17387:2008)

Deliverable D3.2


[21994] Passenger cars – Stopping distance at straight-line braking with ABS – Open-loop

test method (ISO 21994:2007, IDT)

[4138] Passenger cars — Steady-state circular driving behaviour – Open-loop test methods

(ISO 4138:2004).

B.1.3 Definitions

Clearance – Distance from the target vehicle’s trailing surface to the subject vehicle’s

leading surface [15622].

Closed loop – The closed loop method is based on the interaction of a human driver with

the vehicle. In such case, the response from the driver to the actuation from the subject

vehicle and the actual influence of the HMI design is considered as well.

Closing Speed – The closing speed of a target vehicle is defined as the difference between

the target vehicle’s speed and the subject vehicle’s speed. This definition applies to target

vehicles in the rear zone only. A positive closing speed indicates that the target vehicle is

closing on the subject vehicle on the rear [17387].

Collision speed – Collision speed is the relative speed at which the subject vehicle collides

into the target vehicle. The vehicle speeds are measured at the moment of the collision.

Departure – situation in which the outside of one of the front wheels of a vehicle or of the

leading part of an articulated vehicle – or, in the case of a three wheeled vehicle, the outside

of one of the wheels on the axle with the widest track – is crossing a specified line [17361].

Driver intention following – This parameter represents the performance of a vehicle in

terms of how closely it responds to the driver input.

Driver steering input – It will analyse the steering wheel angle, speed, and torque done by

the driver in order to quantify the amount of driver activity.

Electronic stability control (ESC) – ESC is a computerized technology that improves the

safety of a vehicle's stability by detecting and minimizing skids. When ESC detects loss of

steering control, it automatically applies the brakes to help "steer" the vehicle where the

driver intends to go. Braking is automatically applied to individual wheel, such as the outer

front wheel to counter oversteer or the inner rear wheel to counter understeer. Some ESC

systems also reduce engine power until control is regained. ESC does not improve a

vehicle's cornering performance; it rather helps minimize the loss of control.

Lane boundary – borderline of the lane, situated at the centre of a visible lane marking or, in

the absence of a visible lane marking, determined by incidental visible road features or other

means such as GPS, magnetic nails, etc. [17361].

Lane departure – point of departure across the lane boundary [17361].

Deliverable D3.2


Lateral acceleration – The lateral acceleration is the acceleration value measured in the y

direction at the centre of gravity.

Lateral clearance – the lateral clearance of a target vehicle is defined as the lateral distance

between the side of the subject vehicle and the near side of a target vehicle [17387].

Lateral displacement – The lateral displacement is the lateral distance difference between

the start and end points of the manoeuvre.

Local time reference – The local time corresponds to the time reference of the data logger

used during testing.

Longitudinal acceleration – The longitudinal acceleration is the acceleration value

measured in the x direction at the center of gravity.

Longitudinal speed – Speed measured in the x direction at the centre of gravity. Can also

be calculated as the numeric derivative of the local x position with respect to the time.

Open loop – The open loop method is a test without considering the natural response and

feedback from the driver. The open loop test may be performed with a professional driver or

a robot driver.

Oversteer – A condition in which the vehicle’s yaw rate is greater than the yaw rate that

would occur at the vehicle’s speed as result of the Ackerman Steer Angle.

Peak lane deviation – The peak lane deviation is the maximum magnitude of lateral position

as measured from a point on the centreline of the vehicle projected downward to the

pavement and subtracted from lane centreline or other fixed reference.

Rate of departure – Subject’s vehicle approach speed at a right angle to the lane boundary

at the warning issue point [17361].

Rollover condition – Corresponds to the situation where both inner wheels lift more than 5

cm from the ground and for more than 20 ms.

Sideslip or side slip angle – The arctangent of the lateral speed of the centre of gravity of

the vehicle divided by the longitudinal speed of the centre of gravity.

Sideslip angle peaks – They are the minimum and maximum values of sideslip angles

during a manoeuvre. They are stability parameters directly related to the oversteer tendency

of the vehicle.

Steering activity – The steering activity is the integral of the steering wheel angle during the

manoeuvre.

Steering wheel angle average – The steering wheel angle average is the mean value of the

steering wheel angle during the manoeuvre.

Deliverable D3.2


Steering wheel torque peak – The steering wheel torque peak is the absolute value and

sign of the steering wheel torque peak during the manoeuvre.

Standard deviation of lane position – During the manoeuvre, the middle of the lane is

considered as optimum and deviations from the optimum are measured each time step.

Then, the standard deviation of these measures is calculated.

Stopping distance – The stopping distance is the distance travelled by the vehicle during

the braking manoeuvre until complete stop.

Time to collision (TTC) – The time to collision gives clear information on the remaining time

before a collision.

It is calculated by: TTC = longitudinal distance / relative speed

Time to line crossing (TLC) – This value represents the time remaining before crossing the

line. The objective is to know how much time before (or after) crossing the line the system

warns or acts.

TLC = lateral distance to lane / lateral speed

Trial – A single execution of a test case.

Understeer – A condition in which the vehicle’s yaw rate is less than the yaw rate that would

occur at the vehicle’s speed as a result of the Ackerman Steer Angle.

Use of adherence – A parameter that quantifies the characteristics of the vehicle for the

available adherence in μ-split scenario.

Validation – Validation describes the process of evaluating the system impact e. g. on

safety. That is, validation checks and tests whether the system "does what it was designed

for", e.g. increase traffic safety by increasing headway, by avoiding impacts and so on. For

this purpose the driver needs to be in the loop. Driver in the loop testing is required.

Vehicle reference coordinate system – According to the common convention for the

vehicle coordinates system (ISO 8855), the x axis points to the front of the vehicle, the y axis

points to the left and the z axis points upwards (right-hand orthogonal system).

Vehicle speed variation – It is the difference in speed between the start and the end of the

manoeuvre. Depending on the function/manoeuvre and together with sideslip angles peaks,

it gives an idea about how much the ESC has to work in order to keep the vehicle stable.

Verification – Verification describes the test of a system/ function against its requirements,

that is, whether it fulfils its requirements.

Warning issue point – measured location and time at which a warning starts to be issued.

Deliverable D3.2


Warning threshold – location where the warning is issued on the road, which corresponds

to a warning trigger point set in the system. The warning threshold shifts depending on the

rate of departure. The warning threshold is placed within the warning threshold placement

zone [17361] (see Figure B-1).

Warning threshold placement zone – zone between the earliest and the latest warning

lines within which the warning threshold is placed. There is a warning threshold placement

zone around the left lane boundary and one around the right lane boundary (see Figure B-1).

14

53

2

2

6

1 lane boundary 4 latest warning line

2 warning threshold placement zone 5 no warning zone

3 earliest warning line 6 warning threshold (for reference only)

Figure B-1 Concept of warning thresholds and warning threshold placement zone [17361]

Yaw response – The yaw response is the rotation around the z at the centre of gravity.

Yaw rate – The rate of change of the vehicle’s heading angle measured in degree/second of

rotation about the vertical axis through the vehicle’s centre of gravity.

Yaw rate ratios – They are the ratios between the yaw rate at certain times after the end of

the steering wheel actuation and the yaw rate peak. They are the stability parameters

suggested by NHTSA ESC NCAP Confirmation Test.

B.1.4 Test conditions and environment

The environmental conditions shall not change during the course of the test. Allowed

variations for temperature, visibility, and wind are stated below.

Deliverable D3.2


B.1.4.1 Test track conditions

All tests shall be carried out on a smooth, clean, dry (except for µ-split tests) and uniform

paved road surface with local lane markings. The friction coefficient of the road surface shall

not differ from standard road types1. The gradient of the paved test surface to be used sall

not exceed 2% (recommended 1.5%) in any direction when measured over any distance

interval between that corresponding to the vehicle track and 25 m [15037].

The track shall have an area where lanes with a radius of curvature of up to 500 m ±10% are

provided.

In testing protocols where lane width is important, a minimum lane width of 3.5 m shall be

considered [ECE 384].

Dashed lane markings shall be used during the lane departure tests. Different lane marking,

in terms of e.g. white coverage, applies to different European countries. So far no

standardized dashed lane marking have been decided for lane departure testing

B.1.4.2 Temperature conditions

The ambient temperature shall be between 0 ºC and 40 ºC and its variation during a

sequence of measurements shall be below 10 ºC.

The test track surface temperature shall be between 0 ºC and 50 ºC and its variation during a

sequence of measurement shall be below 10 ºC.

B.1.4.3 Visibility conditions

All tests are to be conducted during daytime. The horizontal visibility shall be greater than 1

km [15623].

B.1.4.4 Wind conditions

The ambient wind speed (regardless of wind direction) shall either not exceed 3 m/s or, if the

wind speed ranges between 3 m/s and 5 m/s maximum, an equal number of measurement

specified shall be carried out in both driving directions [21994].

B.1.5 Subject vehicle preparation and conditioning

The condition of the subject vehicle shall be in accordance with the vehicle manufacturer’s

specifications [21994].

1 Concerning the split tests, the friction coefficient of the high test surface shall be between 0.8 and 1.0 and its variation

shall not exceed ±5% over the length of the test surface. The friction coefficient of the low test surface shall be between 0.10

and 0.2, and its variation shall not exceed ±5% over the length of the test surface.

Deliverable D3.2


For cluster 1 and cluster 3 tests, tyres and brakes shall be conditioned according to ISO

21994 [21994].

B.1.5.1 Data collection systems

The subject vehicle is to be equipped with measurement equipment in order to record

measures and to warranty their precision and to fulfil the repetitiveness requirements for the

tests. The onboard measurement system shall provide a data collection from which it is

possible to extract the measures needed for computation of safety performance indicators.

The IMU (inertial measurement unit) sensor shall be located as close to the centre of gravity

of the vehicle as possible. In any case measured signals must be compensated of sensor

position effects and carried to the centre of mass.

Optionally, video recorders for collecting video sequences of the road scene and driver shall

be employed. Preferably, the video recording shall be synchronised with the data collecting

system. Driver and lane video sequences shall, if possible, be sampled at the same

frequency as other sensors (≥100 Hz) but 30 Hz might be sufficient.

B.1.5.2 Configuration of subject vehicle

The combined mass (of driver and test equipment) should not exceed 150 kg [15037].

The fuel tank shall be full and, in the course of the measurement sequence, the indicated fuel

level should not drop below ―half-full‖ [21994].

If the safety systems provide different settings, the default setting shall be used.

The tires used shall be standard ones as recommended by the OEM and with the

recommended pressure.

Deliverable D3.2


B.2 Avoidance or mitigation of rear-end collision, open loop test

B.2.1 Scope

This chapter is the testing protocol that describes test procedures for testing the performance

of the longitudinal safety functions in a scenario where a vehicle is approaching another

vehicle which is moving slower in the same direction, decelerating, or being stationary on a

straight or curved road. The present test procedure addresses the safety of different types of

subject vehicles (passenger vehicle, bus or commercial vehicle).

The different tests shall represent different driver behaviour in longitudinal accident-related

scenarios.

The test procedures consider physical tests for evaluating the safety performance of the

vehicle by using professional drivers and/or driving robots that control the vehicle with

predefined manoeuvres. Warning, support, and mitigation functions are expected to be

activated during the manoeuvres.

The test procedure determines the safety performance indicators during the driving

manoeuvres.

Subject vehicle Target vehicle

Wt

at , vtas, vs

Subject vehicle

Target vehicle vt

as , vs

Figure B-2 Rear-end straight and curved road scenarios

B.2.2 References

References are listed in Chapter B.1.

B.2.3 Definitions

Definitions are listed in Chapter B.1.

Deliverable D3.2


B.2.4 Test procedure - Avoidance or mitigation of rear-end collision

B.2.4.1 Principle

The test is based on the observation of the subject vehicle behaviour when executing the

manoeuvres specified in respective test. The open loop tests are focusing on the vehicle’s

technical performance.

The manoeuvres on the test tracks shall resemble real accident-prone traffic situations.

B.2.4.2 Test objectives

The objective of an open loop test is to test the technical performance of the vehicle, without

considering natural response and feedback from an arbitrary driver. A professional driver or a

driving robot is used for triggering an action from the vehicle. The timing sequence of the

open loop tests is presented in Figure B-3. There are different open loop tests depending on

the type of action from the professional driver or driving robot (no action, mild brake after

warning, and strong brake after warning). In test 1, the driver is passive, whereas in tests 2

and 3 the driver brakes strongly and mildly, respectively. The outcome of the tests will

depend on the level of assistance from the subject vehicle (warning, support, and/or

intervention).

t0 t1 t3 t4 t5 t6

Initial conditions for

subject vehicle and target

vehicle velocity and

clearance fulfilled

Pre-stabilisation

period

Warning Support

Collision

No Collision

Start of test

Intervention

Collision

No Collision

Collision

No Collision

Strong

braking action

Mild braking

action

Typical driver

reaction time

tr

Test 1

Test 2

Test 3

t2

Possible brake

by target vehicle

Passive

Figure B-3 Time sequence for open loop tests

Deliverable D3.2


B.2.4.3 Drivers

Professional drivers or preferably driving robots are needed.

B.2.4.4 Equipment

Besides the equipment used to log positions, speeds, and accelerations (e.g. GPS with IMU)

of the subject and target vehicle respectively, there is a need to synchronously log warnings

provided from the vehicle to the driver. The warning signals might be obtained by tapping the

CAN bus, but preferably an external sensor should be used (e.g. a photo-sensitive device for

visual warnings). The collision instant can either by registered by a pressure-sensitive sensor

mounted at the front of the subject vehicle, or by registering the instant at which the distance

(clearance) between the two vehicles becomes zero. The distance between subject and

target vehicle can be calculated from synchronized high-precision GPS position data or be

directly measured using a RADAR or LASER distance sensor.

B.2.4.4.1 Target vehicle

To facilitate a possible collision, the target vehicle is simulated by a vehicle dummy similar to

ordinary vehicles with regard to physical dimensions and detection characteristics. The

following properties are important1:

- Size (width, height, and shape)

- Radar Cross Section (RCS)

- Reflectivity

- Size, contrast, and colour range

B.2.4.5 Testing environment

Test conditions and environment are described in Chapter B.1.

B.2.4.6 Information required for the test

The resulting reports from the following inspections are used.

- Definition of subject vehicle including the targeted safety functions

- Environmental conditions

These reports are necessary to understand if normal conditions are applicable during the

physical tests. Also these inspections help to get an understanding of the subject vehicle and

its safety functions.

1 Target vehicle dimensions and characteristics are still under discussion; eVALUE will not define a general target

vehicle. However, the generic target being developed in the ASSESS project might be a suitable candidate.

Deliverable D3.2


B.2.4.7 Subject vehicle preparation

Subject vehicle preparation and conditioning are described in Chapter B.1.

B.2.4.8 Test procedure and data processing

B.2.4.8.1 Tests

The three tests in this testing protocol are:

Test 1: Rear-end collision – passive driver

Test 2: Rear-end collision – driver brakes strongly

Test 3: Rear-end collision – driver brakes mildly

For each of the three tests there a number of test cases. The test cases represent different

combinations of subject vehicle speed as well as target vehicle speed and deceleration.

Additionally the test cases consider: straight road, left curve, or right curve.

During all the tests, the measurements listed in Table B-1 shall be logged. From these

measurements, the safety performance indicators can be directly determined or calculated.

Table B-1 Measurements to be recorded during the tests

Measure Typical range Recommended

maximal error

local time reference 10 ms samples

local position of both vehicles 0 m to 250 m ± 0.50 m [21994]

speed of both vehicles 0 km/h to 100 km/h ± 0.5 km/h [21994]

longitudinal deceleration of both vehicles 1 m/s2 to 12 m/s2

0.1 g to 1.2 g

± 0.15 m/s2 [15037]

± 0.015 g

longitudinal distance between both vehicles

0 m to 250 m ± 1% (≤ 50 m)

± 0.50 m (> 50 m)

[21994]

lateral distance between vehicles 30 to + 30 m ± 0.05 m

warning instant N/A video: ±0.05s

audio: ±0.05s

collision instant (if there is any) N/A ±0.05s

brake pedal actuation force 0 – 1000 N ±2% [21994]

Tests 1 to 3 presented in the time sequence diagram in Figure B-3 shall be performed

whenever relevant for the function considered. Regardless of test, the subject vehicle follows

the target vehicle in the same driving lane. During the entire test, the alignment of both

vehicles’ longitudinal axis shall not deviate by more than ± 0.25 m.

Deliverable D3.2


After the pre-stabilisation period, t1, the initial speeds (and clearance) has been established

by the use of professional drivers in the subject and target vehicles. Depending on the test

case, the target vehicle may initiate a robot-controlled braking at t2 (t2- t1 = 3 s).

Subsequently, typical driver action is simulated by doing nothing (passive driver) when the

warning is issued or by a robot-controlled braking after a typical reaction time has elapsed.

The tests progress until a collision occurs or when the speed of the subject vehicle is equal

or lower than that of the target vehicle, i.e. no collision in Figure B-3.

Technical requirements and safety performance indicators

Measurement data is collected during the tests to verify that the test was correctly executed

(e.g. that initial speed was within tolerances) or computing the values of the safety

performance indicators. The technical requirement and safety performance indicators,

reflecting the performance of the vehicle, are presented in Table B-2.

The safety performance indicator with a direct and strong connection to safety is the collision

speed. A reduced collision speed reduces the collision energy and consequently causes less

harm to the subject vehicle occupants. The lower collision speed, the safer it is.

A second safety performance indicator, which indirectly affects safety, is the TTC at warning

(for TTC calculations, constant target speed or deceleration is assumed). If the TTC is too

large, false alarms will be the result, and the driver might deactivate the safety system, which

has an adverse effect on safety. Thus there is an upper bound for this safety performance

indicator. This bound is dependent on the vehicle speed.

Table B-2 Technical requirements and safety performance indicators for open loop tests

Technical requirement Safety performance indicator

The subject vehicle shall avoid a collision

or reduce the collision speed at a potential

rear-end collision

collision speed1

The subject vehicle shall not produce

nuisance alarms

time-to-collision at warning

1 The collision speed must be related/adjusted to the maximal deceleration available for the particular

track-tyres combination.

Deliverable D3.2


B.2.4.8.2 Test 1: Rear-end collision – passive driver

Description

In this test a completely passive driver is simulated. After the initial speeds and clearance

have been established, the subject vehicle driver does nothing until the end of the test

(collision or no collision).

The test shall be performed with the initial speeds, clearances, decelerations according to

the values listed in Table B-3. The curve radius should be 250 m and the inner edge of the

curve shall be shielded with a fence or similar to limit the field of view.

Table B-3 Test cases – passive driver

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial1

target

vehicle

speed

[km/h]

Initial clearance

between

subject and

target vehicle

[m]

Target vehicle

deceleration

[m/s2]

Road

topology

1.1 50±2 0 150±0.5 0±0.5 straight

1.2 70±2 0 150±0.5 0±0.5 straight

1.3 70±2 30±2 100±0.5 0±0.5 straight

1.4 70±2 50±2 100±0.5 0±0.5 straight

1.5 70±2 70±2 40±0.5 3±0.5 straight

1.6 70±2 70±2 40±0.5 5±0.5 straight

1.7 50±2 0 150±0.5 0±0.5 left curve

1.8 50±2 0 150±0.5 0±0.5 right curve

1.9 70±2 30±2 100±0.5 0±0.5 left curve

1.10 70±2 30±2 100±0.5 0±0.5 right curve

1.11 70±2 70±2 40±0.5 5±0.5 left curve

1.12 70±2 70±2 40±0.5 5±0.5 right curve

The number of trials for each test case is ≥1.2

1 For the test cases with a slower moving target vehicle (TC1.3-1.4 and 1.9-1.10) the initial speed of

the target vehicle shall be kept within the limits throughout the test manoeuvre.

2 The number of trials has not yet been decided. More testing and evaluation of the testing protocols

are needed. Please refer to eVALUE deliverable D4.2 for a discussion on this issue.

Deliverable D3.2


B.2.4.8.3 Test 2: Rear-end collision – driver brakes strongly

Description

In this test a driver who brakes strongly is simulated. After the initial speeds and clearance

have been established, the brake action is controlled by a brake robot to warrant

repeatability. The brake action is triggered by the warning signal from the vehicle. A typical

reaction time of 1 s is waited before the robot applies a brake force to the brake pedal. In this

case a pedal force of 700 N should be applied in 0.2 s and kept during the braking action.




Table B-4 Test cases – driver brakes strongly

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial1

target

vehicle

speed

[km/h]

Initial clearance

between

subject and

target vehicle

[m]

Target vehicle

deceleration

[m/s2]

Road

topology

2.1 50±2 0 150±0.5 0±0.5 straight

2.2 70±2 0 150±0.5 0±0.5 straight

2.3 70±2 30±2 100±0.5 0±0.5 straight

2.4 70±2 50±2 100±0.5 0±0.5 straight

2.5 70±2 70±2 40±0.5 3±0.5 straight

2.6 70±2 70±2 40±0.5 5±0.5 straight

2.7 50±2 0 150±0.5 0±0.5 left curve

2.8 50±2 0 150±0.5 0±0.5 right curve

2.9 70±2 30±2 100±0.5 0±0.5 left curve

2.10 70±2 30±2 100±0.5 0±0.5 right curve

2.11 70±2 70±2 40±0.5 5±0.5 left curve

2.12 70±2 70±2 40±0.5 5±0.5 right curve






Deliverable D3.2


B.2.4.8.4 Test 3: Rear-end collision – driver brakes mildly

Description

In this test a driver who brakes strongly is simulated. After the initial speeds and clearance



reaction time of 1.5 s is waited before the robot applies a brake force to the brake pedal. In

this case a pedal force of 350 N should be applied in 0.4 s and kept during the braking

action.




Table B-5 Test cases – driver brakes mildly

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial1

target

vehicle

speed

[km/h]

Initial clearance

between

subject and

target vehicle

[m]

Target vehicle

deceleration

[m/s2]

Road

topology

3.1 50±2 0 150±0.5 0±0.5 straight

3.2 70±2 0 150±0.5 0±0.5 straight

3.3 70±2 30±2 100±0.5 0±0.5 straight

3.4 70±2 50±2 100±0.5 0±0.5 straight

3.5 70±2 70±2 40±0.5 3±0.5 straight

3.6 70±2 70±2 40±0.5 5±0.5 straight

3.7 50±2 0 150±0.5 0±0.5 left curve

3.8 50±2 0 150±0.5 0±0.5 right curve

3.9 70±2 30±2 100±0.5 0±0.5 left curve

3.10 70±2 30±2 100±0.5 0±0.5 right curve

3.11 70±2 70±2 40±0.5 5±0.5 left curve

3.12 70±2 70±2 40±0.5 5±0.5 right curve






Deliverable D3.2


B.2.4.9 Uncertainty

Uncertainty in the measurement data shall be stated.

B.2.4.10 Result

The measurements and the safety performance indicators associated to the subject vehicle

for the current test procedure shall be documented in the test result. The values shall be

listed together with their respective uncertainty.

The execution of the test procedure shall be briefly described.

The test vehicle shall be identified with reference to the corresponding inspection.

Deliverable D3.2


B.3 Avoidance of collision with transversally moving target, open loop test

B.3.1 SCOPE

This testing protocol describes the test procedure for testing the performance of the

longitudinal safety functions of a vehicle approaching a transversally moving target. The

moving target is a passenger vehicle, a motorcycle or a pedestrian. The present test

procedure addresses the safety of different types of subject vehicles (car or specific types of

commercial vehicles).

The different tests shall represent different driver behaviours in longitudinal accident-related

scenarios.

The test procedures consider physical tests for evaluating the safety performance of the

vehicle by using professional drivers and driving robots that control the vehicle with

predefined manoeuvres. Warning, support, and mitigation functions are expected to be

activated during the manoeuvres.

The test procedure determines the safety performance indicators during the driving

manoeuvres.

Subject vehicle

Target vehicle

vt

vs

Figure B-4 Transversally moving target scenario

B.3.2 References


B.3.3 Definitions


B.3.4 Test procedure - Avoidance of collision with transversally moving target

B.3.4.1 Principle


manoeuvres specified in respective test. The open loop tests are focusing on the vehicle’s

technical performance.

Deliverable D3.2


The manoeuvres on the test tracks shall resemble real traffic situations.




driving robot is used for triggering an action from the vehicle. The timing sequence of the

open loop tests is presented in Figure B-5. There are different open loop tests depending on

the type of action from the professional driver or driving robot (no action, mild brake after

warning, and strong brake after warning). In test 1, the driver is passive, whereas in tests 2

and 3 the driver brakes strongly and mildly, respectively. The outcome of the tests will

depend on the level of intervention from the subject vehicle (warning, support, and/or

intervention).

t0 t1 t3 t4 t5 t6

Initial conditions for

subject vehicle and target

vehicle velocity and

clearance fulfilled

Pre-stabilisation

period

Warning Support

Collision

No Collision

Start of test

Intervention

Collision

No Collision

Collision

No Collision

Strong braking action

Mild braking action

Typical driver

reaction time

tr

Test 1

Test 2

Test 3

Passive

Figure B-5 Time sequence for open loop tests

B.3.4.3 Drivers

Professional drivers or preferably driving robots are needed.

B.3.4.4 Equipment

Besides the equipment used to log positions, speeds, and accelerations (e.g. GPS with IMU)

of the subject and target vehicle respectively, there is a need to synchronously log warnings

provided from the vehicle to the driver. The warning signals shall preferably be detected by

an external sensor (e.g. a photo-sensitive device for visual warnings). If provided by the

Deliverable D3.2


OEM, warning signals may be obtained via the CAN bus. The collision instant can either by

registered by a pressure-sensitive sensor mounted at the front of the subject vehicle, or by

registering the instant at which the distance (clearance) between the two vehicles becomes

zero. The distance between subject and target vehicle can be calculated from synchronized

high-precision GPS position data.


To facilitate a possible collision, the target vehicle is simulated by a vehicle dummy similar to

ordinary vehicles with regard to physical dimensions and detection characteristics. The

following properties are important1:

- Size (length, height, and shape)


- Reflectivity


B.3.4.4.2 Pedestrian target

The dummy target shall have characteristics similar to a real person in size and movement

and movement speed.



An urban environment where the lateral field-of-view is restricted by buildings shall be used.





These reports are necessary to understand if normal conditions are applicable during the






vehicle. However, the generic target being developed in the ASSESS project might be a suitable candidate.

Deliverable D3.2



B.3.4.8.1 Tests

The three tests in this testing protocol are:

Test 1: Transversally moving target – passive driver

Test 2: Transversally moving target– driver brakes strongly

Test 3: Transversally moving target– driver brakes mildly

For each of the three tests there a number of test cases. The test cases represent different

combinations of subject and target vehicle speeds as well as initial distances. Also there are

test cases for different target objects: passenger vehicle and pedestrian.

During all the tests, the measurements listed in Table B-6 shall be logged. From these the

safety performance indicators can be directly determined or calculated.

Table B-6 Measurements to be recorded during the tests


maximal error

local time reference 10 ms samples

local position of subject vehicle and target object

0 m to 250 m ± 0.50 m [21994]

speed of subject vehicle and target object

0 km/h to 100 km/h ± 0.5 km/h [21994]

longitudinal deceleration 1 m/s2 to 7 m/s2

0.1 g to 0.7 g

± 0.15 m/s2 [15037]

± 0.015 g

longitudinal distance between subject vehicle and target object

0 m to 300 m ± 0.05 m

lateral distance between subject vehicle and target object

-150 m to + 150 m ± 0.05 m

warning instant N/A video: ±0.05s

audio: ±0.05s

collision instant (if there is any) N/A ±0.05s

brake pedal actuation force 0 – 1000 N ±2% [21994]

Tests 1 to 3 presented in the time sequence diagram in Figure B-5 shall be performed. The

subject vehicle enters an intersection and a target vehicle/object crosses its path by a

transversal movement.

To successfully perform the tests, the positions and speeds of the subject vehicle and target

object must be controlled with very small errors. Communication between the subject and

target is also necessary to synchronize their movements.

Deliverable D3.2


After the pre-stabilisation period, t1, the initial speeds and distances has been established by

the use of robot drivers in the subject and target vehicles.

Subsequently, typical driver action is simulated by doing nothing (passive driver) when the

warning is issued or by a robot-controlled braking after a typical reaction time has elapsed.

The tests progress until a collision occurs, or when the target vehicle/object is out of the path

of the subject vehicle, i.e. no collision in Figure B-5.

Technical requirements and indicators

Measurement data is collected during the tests to verify that test was correctly executed (e.g.

that initial speeds were within tolerances), or computing the values of the safety performance

indicators. The technical requirement and safety performance indicators, reflecting the

performance of the vehicle, are presented in Table B-7.

The safety performance indicator with a direct and strong connection to safety is the collision

speed. A reduced collision speed reduces the collision energy and consequently causes less

harm to the subject vehicle occupants. The lower collision speed, the safer it is.

A second safety performance indicator, which indirectly affects safety, is the TTC at warning

(for TTC calculations, constant target speed or deceleration is assumed). If the TTC is too

large, false alarms will be the result, and the driver might deactivate the safety system, which

has an adverse effect on safety. Thus there is an upper bound for this safety performance

indicator. This bound is dependent on the vehicle speed.


Technical requirement Safety performance indicator

The subject vehicle shall avoid a collision

or reduce the collision speed with a

transversally moving target.

collision speed1

The subject vehicle shall not produce

nuisance alarms

time-to-collision at warning

B.3.4.8.2 Test 1: Transversally moving target – passive driver

In this test a completely passive driver is simulated. After the initial speeds and distances

have been established, the subject vehicle driver does nothing until the end of the test

(collision or no collision).

1 The collision speed must be related/adjusted to the maximal deceleration available for the particular

track-tyres combination.

Deliverable D3.2


The test shall be performed with the initial speeds and distances according to the values

listed in Table B-8. Figure B-6 shows the definitions of initial speeds and distances. Both a

passenger vehicle and pedestrian target shall be used.

V T

LS

Targetvehicle

V S

Subject

vehicle

T L

Subject

vehicle

VS

LS

LT

Pedestrian

target

VT

Figure B-6 Initial conditions

Table B-8 Test cases – passive driver

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial

target

object

speed

[km/h]

Initial

subject

vehicle

distance (LS)

[m]

Initial

target

object

distance

(LT)

[m]

Target object

1.1 30±2 15±2 30±1 15±0.25 passenger vehicle



1.4 15±2 5±0.5 15±1 5±0.25 pedestrian

1.5 30±2 5±0.5 30±1 5±0.25 pedestrian

1.6 50±2 5±0.5 50±1 5±0.25 pedestrian


B.3.4.8.3 Test 2: Transversally moving target - driver brakes strongly

In this test a driver who brakes strongly is simulated. After the initial speeds and distances





Deliverable D3.2


reaction time of 1 s is waited before the robot applies a brake force to the brake pedal. In this

case a pedal force of 700 N in 0.2 s should be applied.


listed in Table B-9. Both a passenger vehicle and pedestrian target shall be used.

Table B-9 Test cases – driver brakes strongly

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial

target

object

speed

[km/h]

Initial

subject

vehicle

distance (LS)

[m]

Initial

target

object

distance

(LT)

[m]

Target object




2.4 15±2 5±0.5 15±1 5±0.25 pedestrian

2.5 30±2 5±0.5 30±1 5±0.25 pedestrian

2.6 50±2 5±0.5 50±1 5±0.25 pedestrian


B.3.4.8.4 Test 3: Transversally moving target – driver brakes mildly

In this test a driver who brakes strongly is simulated. After the initial speeds and distances



reaction time of 1.5 s is waited before the robot applies a brake force to the brake pedal. In

this case a pedal force of 350 N in 0.4 s should be applied.


listed in Table B-10. Both a passenger vehicle and pedestrian target shall be used.



Deliverable D3.2


Table B-10 Test cases – driver brakes mildly

Test

case

Initial

subject

vehicle

speed

[km/h]

Initial

target

object

speed

[km/h]

Initial

subject

vehicle

distance (LS)

[m]

Initial

target

object

distance

(LT)

[m]

Target object




3.4 15±2 5±0.5 15±1 5±0.25 pedestrian

3.5 30±2 5±0.5 30±1 5±0.25 pedestrian

3.6 50±2 5±0.5 50±1 5±0.25 pedestrian


B.3.4.9 Uncertainty


B.3.4.10 Result








Deliverable D3.2


B.4 Lane departure, open loop test

B.4.1 Scope

This testing protocol describes the procedure for evaluating the performance of the safety

functions for avoidance of lane departure on a straight or curved road. The present test

procedure addresses the safety of different types of subject vehicles (passenger car, bus or

truck).

The test procedures define physical tests for evaluating the safety performance of the vehicle

by using professional drivers or driving robots that control the vehicle with predefined

manoeuvres.

This testing protocol describes open loop tests.

vt

vt

Figure B-7 Illustration of lane departure scenarios on a straight road.

B.4.2 References


B.4.3 Definitions


B.4.4 Test procedure - Avoidance of lane departure

B.4.4.1 Principle


manoeuvres specified in the respective tests 1 to 3, corresponding to scenarios C2-1 to C2-3

as stated in Chapter 2.2. The open loop tests address the vehicle’s technical performance.

For the present tests no distinction is made between lane departure and road departure, i.e.

the former is considered to cover the latter.

Deliverable D3.2


The test itself aims at determining the subject vehicle’s performance by warning the driver to

prevent the occurrence of a lane departure. The following safety performance indicator for

the subject vehicle is determined:

- time to line crossing (TLC) at warning


This test protocol defines tests for open loop testing where a professional test driver or

driving robot is used.



driving robot is used to drive the car in such way that a warning should be triggered. The

timing sequence of the open loop test is presented in Figure B-8.

t0 t1 t3 t4t2

Pre-

stabilisation

period

Create a lane

drift

Await

system

activation

(warning)

Start of test Lane drift established

and stable

Warning End of test

Figure B-8 Timing sequence for open loop testing of lane departure

B.4.4.3 Drivers

A professional driver or a driving robot is needed in the subject vehicle.

B.4.4.4 Equipment

The equipment used for the test procedure shall provide for the collection of appropriate

measures and their processing into safety performance indicators. A positioning system shall

be able to determine the position of the subject vehicle throughout the entire test. The time

for issuing warning signals in the subject vehicle must be accessible and possible to

synchronize with the subject vehicle position.

B.4.4.5 Testing conditions and environment



The resulting reports from the following inspections shall be used:

Deliverable D3.2


- Definition of subject vehicle including the targeted safety functions.

o In particular, it is important to determine the activation speed of the safety

system and whether its design and implementation can be considered

symmetric, such that equal performance can be expected for both sides of the

vehicle.


These reports are necessary to understand if e.g. normal conditions can be used during the

physical tests and to get an understanding of the subject vehicle and its safety functions.

An accurate position reference of the lane markings is needed. The accuracy should be at

least equal to the subject vehicle position accuracy, cf. Table B-11. In addition, evaluation of

line crossing requires that the vehicle dimensions as well as the placement of the antenna (or

similar) of the positioning system are known.

B.4.4.7 Subject vehicle preparation and conditioning



B.4.4.8.1 Tests

The test procedure addresses lane (and road) departure on a straight or curved road

according to the following tests:

Test 1 Lane departure on a straight road (Figure B-9)

Test 2 Lane departure in a curve (Figure B-10)

Test 3 Lane departure on a straight road just before a curve (Figure B-11)

During the course of the tests, the measures listed in Table B-11 are to be recorded.

Deliverable D3.2


Table B-11 Measurements during lane departure tests


maximal error

position of subject vehicle

0 m to 750 m X,Y: ± 0.05m

speed of the subject vehicle

0 km/h to 110 km/h ± 0.5 km/h [21994]

rate of departure - 5 m/s to 5 m/s ± 0.05 m/s

local time reference

shall cover all test duration, from stabilization

until lane avoidance or crossing, (typically 20 s)

and apply to all measured quantities.

video: ±0.01s

haptic: ±0.01s

audio: ±0.01s

subject vehicle

deviation from

the lane

boundary

0 m to 5 m X,Y: ± 0.05m

point of departure

across the lane

boundary

0 m to 250 m X,Y: ± 0.05m

warning signal (to the driver) issuing time

extracted from driver/dashboard video for visual

signals, audio warnings will be detected by using

microphones, haptic warnings by appropriate

sensors (ex accelerometers)

video: ±0.05s

haptic: ±0.05s

audio: ±0.05s

The measured values are logged during the driving manoeuvres until termination of the test.

Instructions about the manoeuvres to be carried out are given to the test driver of the subject

vehicle. At the end of the pre-stabilisation period, the subject vehicle reaches the initial speed

and moves along the centre line of the lane with the lateral safety function in operational

mode. The subject vehicle is guided to drift and eventually depart from the lane in a

predefined manner, either programmed into a steering robot or specified as a deviation zone

on the track indicating to the driver the course to follow while maintaining the specified

speed. The test driver is not allowed to correct the path before a warning is given. When a

warning is issued, the end of the test is reached.

Deliverable D3.2



Measurements are carried out to monitor the time for and type of warning provided by the

vehicle given certain predefined driving manoeuvres. The assessment of the safety

performance of the vehicle is based on the time of warning.

The technical requirements and safety performance indicators are presented in Table B-12.


Technical requirements Safety performance indicator

The system shall warn the driver when the

warning condition is fulfilled [17361]

No warning shall occur between the two

earliest warning lines as described in

Section B.1.3 [17361]

time to line crossing (TLC) at warning

The safety performance indicator ―Time to Line Crossing― is determined based on the vehicle’s position and rate of departure (lateral speed) at the time of warning, as well as on the vehicle geometry and the position reference of the lane marking. It is computed in the following way:

- The shortest distance between the lane marking and the vehicle, divided by the rate of departure.

B.4.4.8.2 Test 1: Lane departure on a straight road

Description

vt

vt

Figure B-9 Illustration of lane departure scenarios on a straight road.

The timing sequence for open loop tests is presented in Figure B-8.

Deliverable D3.2


1. During the pre-stabilisation period (t0 - t1), the subject vehicle reaches the predefined

initial speed while moving along the centre line of the lane with the lateral safety

function activated.

2. Keeping the initial longitudinal speed, the driver of the subject vehicle is at a time

between t1 and t2 forced to deviate from the centre line.

3. The test is concluded when

a) the warning is issued or

b) half the vehicle has crossed the line into the adjacent lane (regarded as a

complete failure)

The tests shall be carried out with initial speed, rate of departure and departure direction

according to Table B-13.

Test implementation

The subject vehicle moves in a straight course of 100 m at the specified speed along the

centre of the lane. The lane departure test may be implemented using e.g. a path following

function or in terms of a predefined steering wheel angle, to obtain the predefined rate of

departure (lateral speed) stated in Table B-13 below.

When the pre-stabilization period is finished, the subject vehicle creates a lane drift with the

specified lateral speed.

Once the test is concluded, the lane change manoeuvre should be aborted by the driver.

The following considerations are needed in order to define longitudinal speed, rate of

departure and departure direction, based on inspection of the subject vehicle mentioned in

Section B.4.4.6:

Initial longitudinal speed. A low and a high longitudinal speed for testing shall be

selected to ensure that the lane departure warning system is operational.

o Low longitudinal speed = Minimum system activation speed + 5 km/h, no lower

than 72 km/h.

o High longitudinal speed = Low longitudinal speed + 30 km/h, not exceeding

110 km/h.

Rate of departure. A low and a high departure rate shall be determined based on the

chosen longitudinal speeds in the following way:

o Low departure rate interval = 1% of prescribed longitudinal speed ± 0.1 m/s

o High departure rate interval = 3% of prescribed longitudinal speed ± 0.1 m/s

Departure direction. If it is concluded from the inspection that the system is expected

to perform equally on both sides of the subject vehicle, it is considered sufficient to

choose either left or right as departure direction for all tests. If, in contrast, the system

Deliverable D3.2


cannot be considered as essentially symmetric, the testing of each case needs to be

carried out in both directions.

Table B-13 Test cases for lane departure on a straight road

Test

case

Initial

subject

vehicle

speed

Rate of departure Departure

direction

Lane marking

1.1 low low left/(right) dashed white

1.2 low high left/(right) dashed white

1.3 high low left/(right) dashed white

1.4 high high left/(right) dashed white


B.4.4.8.3 Test 2: Lane departure in a curve

Description

vt

R

Figure B-10 Illustration of a lane departure scenario in a curve

These tests should be carried out in a curve of radius between 300 and 600 m.

The timing sequence for open loop tests is presented in Figure B-8.



function activated



Deliverable D3.2


2. Keeping the initial longitudinal speed, the driver of the subject vehicle is at a time

between t1 and t2 forced to deviate from the centre line.

3. The test is concluded when:

a. the warning is issued

b. half the width of the vehicle leading edge has crossed over the line into the

adjacent lane (regarded as complete failure)

The test shall be carried out with initial longitudinal speed and rate of departure according to

Table B-14.

Longitudinal speeds and rates of departure are equal to those defined for Test 1 in Section

B.4.4.8.2.

Test implementation

The subject vehicle moves along the centre line of the lane. The lane departure test may be

implemented using e.g. a path following function or in terms of a predefined steering wheel

angle, to obtain the predefined rate of departure (lateral speed) stated in Table B-14 below.

When the pre-stabilization period is finished, the subject vehicle creates the lane drift with the

specified lateral speed.

Once the test is concluded, the lane change manoeuvre should be aborted by the driver.

If it is concluded from the inspection that the system is expected to perform equally on both

sides of the subject vehicle, it is considered sufficient to choose either left or right curve for

all tests. If, in contrast, the system cannot be considered as essentially symmetric, the testing

of each case needs to be carried out in both types of curves.

Table B-14 Test cases for lane departure in a curve

Test

case

Initial

subject

vehicle

speed

Rate of

departure

Curve

direction

Departure

direction

Lane marking

2.1 low low left/(right) left/(right) dashed white

2.2 low high left/(right) left/(right) dashed white

2.3 low low left/(right) right/(left) dashed white

2.4 low high left/(right) right/(left) dashed white

The difficulty associated with high precision driving in curves implies two differences

compared to lane departure testing on a straight road:

1. It is recommended to use a steering robot to obtain adequate driving precision.

Deliverable D3.2


2. For safety reasons, curve testing is only carried out at low longitudinal speed

The number of trials for each test case is ≥11.

B.4.4.8.4 Test 3: Lane departure just before a curve

Description

vs

R

Figure B-11 Illustration of lane departure scenario on a straight lane just before a curve.

These tests should be carried out in a curve of radius between 300 and 600 m.

The timing sequence for open loop tests are presented in Figure B-8.



function activated

2. Keeping the initial longitudinal speed, the driver of the subject vehicle drives straight

into the curve.

3. The test is concluded when:

a. the warning is issued

b. half the width of the vehicle leading edge has crossed the line (regarded as

complete failure)

The test shall be carried out with the initial speed listed in Table B-15.

The longitudinal speed is equal to that defined for Test 1 in Section B.4.4.8.2.



Deliverable D3.2


Test implementation

The subject vehicle moves along the centre line of the lane at the predefined speed. The

vehicle continues to go straight ahead when it enters the curve. Once the test is concluded,

the lane departure should be aborted by the driver.

If it is concluded from the inspection that the system is expected to perform equally on both

sides of the subject vehicle, it is considered sufficient to choose either left or right curve for

all tests. If, in contrast, the system cannot be considered as essentially symmetric, the testing

of each case needs to be carried out in both types of curves.

Table B-15 Test case for lane departure on a straight road just before a curve

Test case Initial subject

vehicle speed

Curve direction Lane marking

3.1 low left/(right) dashed white

3.2 low left/(right) dashed white


B.4.4.9 False alarms in Tests 1-3

There are no specific tests dedicated to false alarms. However, in accordance with ISO

17361, a no-warning zone should be defined, within which any issued alarm is considered as

false.

All false alarms occurring during Tests 1-3 shall be reported.

The no-warning zone may be defined by the OEM or system manufacturer, in which case it

will be stated in the inspection protocol. If no other information is available, a no-warning

zone extending to 30 cm inside the lane boundaries on both sides will respect the conditions

for most light and heavy vehicles driving in the centre of a highway lane.

B.4.4.10 Uncertainty




Deliverable D3.2


B.4.4.11 Result






Deliverable D3.2


B.5 Avoidance of lane change collision on a straight road, open loop test

B.5.1 Scope

This testing protocol describes the procedure for evaluating the performance of the safety

functions for avoidance of lane change collisions. The present test procedure addresses the

safety of different types of subject vehicles (passenger car, bus or truck).

The test procedures define physical tests for evaluating the safety performance of the vehicle

by using professional drivers or driving robots that control the vehicle with predefined

manoeuvres.

This testing protocol describes open loop tests.

Subject vehicle

Target vehiclevt

vs

Figure B-12 Illustration of lane change collision scenario

B.5.2 References


B.5.3 Definitions


B.5.4 Test procedure - Avoidance of lane change collision on a straight road

B.5.4.1 Principle


manoeuvres specified in Test 1 corresponding to scenario C2-4 as stated in Chapter 2.2.

The open loop tests address the vehicle’s technical performance.

The test itself aims at determining the subject vehicle’s performance by warning/supporting

the driver, to prevent a lane change collision. The following safety performance indicator is

determined for the subject vehicle:

- Time until target enters Collision Risk Zone at warning

Deliverable D3.2



This testing protocol defines tests for open loop testing where professional test drivers or

driving robots are used.


considering natural response and feedback from the driver. Professional driver or driving

robots are used to drive the vehicles in such a way that a warning should be triggered by the

subject vehicle.

The timing sequence of open loop tests in lane change scenarios is presented in Figure B-

13.

t0 t1 t3t2

Pre-

stabilisation

period

Start of test Warning End of test

t5t4

Perfrorm

actions to

activate safety

functionality

Target enters

Collision Risk

Zone

Target leaves

Collision Risk

Zone

Figure B-13 Timing sequence of open loop tests for lane change collision

B.5.4.3 Drivers

Professional drivers or driving robots shall be used in the subject and target vehicles.

B.5.4.4 Equipment

The equipment used for the test procedure shall provide for the collection of appropriate

measures and their processing into safety performance indicators. Positioning systems shall

be able to determine the position of both vehicles throughout the entire test. Time stamps or

other means of time synchronization of the vehicle positions must be available. The time for

issuing warning signals must be accessible and possible to synchronize with subject vehicle

position.


The target may either be a real vehicle or simulated by a vehicle dummy similar to an

ordinary vehicle with regard to physical dimensions and detection characteristics.

Deliverable D3.2


The following properties are important1:

- Size (width, height, and shape)


- Reflectivity


B.5.4.5 Testing conditions and environment



The resulting reports from the following inspections:

- Definition of subject vehicle including the targeted safety functions.

o In particular, it is important to determine the activation speed of the safety

system and whether its design and implementation can be considered

symmetric such that equal performance can be expected for both sides of the

vehicle.

o A definition of actions required to activate the subject vehicle’s applicable lane

change support (for instance activation of turn indicator and/or attempt a lane

change).

o Definition of a Collision Risk Zone, within which the system is expected to

generate a warning. The Collision Risk Zone may be stated by the OEM/

system manufacturer in terms of an official performance specification, or

chosen according an applicable standard such as ISO 17387.



physical tests and to get an understanding of the subject vehicle and its safety functions.

The definition of the Collision Risk Zone may be chosen differently depending on the nature

of the safety system. For instance, for lane change support systems based on blind spot

detection, a Collision Risk Zone based on the ISO 17387 definition of blind spot zone may be

chosen.

Evaluation of performance may require knowledge of subject and target vehicle geometries

as well as placement of antennas (or similar) of the positioning systems.


vehicle.

Deliverable D3.2





B.5.4.8.1 Tests

The test procedure addresses lane change collision avoidance on a straight road according

to the following test:

Test 1 Lane change collision avoidance on a straight road (Figure B-12)

During the course of this test, the measures listed in Table B-16 shall be recorded.

Table B-16 Measurements during lane change collision warning tests


maximal error

position of subject vehicle 0 m to 750 m X,Y: ± 0.05m

position of target vehicle 0 m to 750 m X,Y: ± 0.05m

absolute distance between subject vehicle and target vehicle, if available

0 m to 250 m X,Y: ± 0.1m

relative lateral distance between subject vehicle and target vehicle, if available

0 m to 250 m X,Y: ± 0.1m

speed of the subject vehicle

0 km/h to 110 km/h ± 0.5 km/h

[21994]

speed of the target vehicle

0 km/h to 110 km/h ± 0.5 km/h

[21994]

local time reference

shall cover all test

duration and applies to all

measured quantities.

video: ±0.01s

haptic: ±0.01s

audio: ±0.01s

warning signal (to the driver) issuing time

extracted from

driver/dashboard video for

visual signals, audio

warnings will be detected

using microphones, haptic

warnings e.g.

accelerometers

video: ±0.05s

haptic: ±0.05s

audio: ±0.05s

The measured values are logged during the driving manoeuvres until termination of the test.

Deliverable D3.2


Instructions about the manoeuvres to be carried out are given to the test driver of the subject

vehicle. At the end of the pre-stabilisation period the subject vehicle reaches the initial speed

and moves along the centre line of the lane taking such actions that the lateral safety

function is activated, while the target vehicle is manoeuvred to pass through the pre-defined

Collision Risk Zone of the subject vehicle.


Measurements are carried out to monitor the time at which a warning is issued by the subject

vehicle.

The technical requirements and safety performance indicators for open loop tests are

presented in Table B-17.

Table B-17 Technical requirements and safety performance indicators


When the safety function is enabled the system shall

detect the target vehicle and issue a warning to

prevent a lane change collision [17387].

time until target enters Collision

Risk Zone (at instance of warning)

The safety performance indicator ―Time to entering Collision Risk Zone― is determined based on the vehicle positions and speeds at the time of warning, as well as geometry of vehicles and Collision Risk Zone. It is computed in the following way: - The distance between the Collision Risk Zone boundary and the leading or trailing edge of the target vehicle, divided by the overtaking speed.

B.5.4.8.2 Test 1: Lane change collision avoidance on a straight road

Description The lane change open loop tests imply that the subject vehicle is overtaking a target vehicle

in the parallel, adjacent lane at speeds specified in the test cases, or that the target vehicle is

overtaking the subject vehicle in the parallel adjacent lane, either to the right or to the left.

The timing sequence for open loop test is presented in Figure B-13.

1. During the pre-stabilisation period (within t0 and t1) the subject vehicle and the target

vehicle reach their predefined initial speeds while moving along the centre line of their

respective lanes..

2. Keeping the initial speed, the driver of the subject vehicle shall take the necessary

actions to activate the safety function (for instance activate the turn indicator and/or

start a lane change attempt).

Deliverable D3.2


3. The vehicles are driven in such a way that the target vehicle enters the Collision Risk

Zone.

4. The test is concluded when the target vehicle has first entered and then completely

exited the Collision Risk Zone.

It shall be noted that some safety systems will require a lane change attempt in order to be

activated. Such systems require extra safety precautions due to the increased collision risk,

e.g. inflatable bump protection or rubber shielding of the target vehicle.

The test shall be carried out with the initial speeds according to the values listed in Table B-

18.

Test implementation

The subject and target vehicles move along the centre of their respective lanes and maintain

their predefined speed according to the test case definition in Table B-18.

If it is concluded in the inspection that the system is expected to perform equally on both

sides of the vehicle, it is considered sufficient to choose either left or right as the side of

target vehicle approach for all tests. If, in contrast, the system cannot be considered as

essentially symmetric, the testing of each case needs to be performed on both sides.

Table B-18 Test cases lane change collision warning

Test case Target vehicle

approach

Subject vehicle

speed

Target vehicle speed

1.1 from behind left/(right) side 65±2 km/h 75±2 km/h

1.2 from behind left/(right) side 65±2 km/h 90±2 km/h

1.3 from front left/(right) side 75±2 km/h 65±2 km/h

1.4 From front left/(right) side 90±2 km/h 65±2 km/h




Deliverable D3.2


B.5.4.9 False alarms in Test 1

There are no specific tests dedicated to false alarms. However, in accordance with ISO

17387, a no-warning zone shall be defined. When the entire target vehicle is located in this

zone, any issued alarm is considered as false.

All false alarms occurring during Test 1 shall be reported.

The following no-warning zones are devised by ISO 17387.

1. In front of a line extending from the subject vehicle’s leading edge

2. Behind a line located 30 m rear of, and parallel to, the subject vehicle’s trailing edge

B.5.4.10 Uncertainty


B.5.4.11 Result






Deliverable D3.2


B.6 Emergency braking on a μ split, open loop test

B.6.1 Scope

This testing protocol describes the test procedure for testing the safety performance of the

subject vehicle during a braking manoeuvre on a μ-split surface. The μ-split surface is such

that the left wheels of the subject vehicle are exposed to a significantly different coefficient of

friction ( than the right wheels. The present test procedure addresses the safety of different

types of subject vehicles (car or different types of commercial vehicles). The test is an open

loop test.

vs

vs

High µ

Low µ

High µ

Low µ

amax

amax

Figure B-14 μ-split braking scenario

B.6.2 References


B.6.3 Definitions


B.6.4 Test procedure - Emergency braking on a μ split

B.6.4.1 Principle


manoeuvres specified in scenario C3-1: emergency braking on a μ split as stated in Chapter

2.2.

The test aims at determining the performance for the subject vehicle with respect to stability

and stopping distance.


This testing protocol defines tests for open loop testing where a professional test driver or

driving robot is used.

Deliverable D3.2


The objective of an open loop test is to evaluate the technical performance of the vehicle,

without considering natural response and feedback from the driver. A professional driver or a

driving robot is used to trigger an actuation from the vehicle.

Timing sequences of open loop tests in the emergency braking on a μ split scenario is

presented in Figure B-15.

Start of test

Velocity

stabilized

Braking manoeuvre

started

Stabilization function active

End of test

t0 t3 t1 t4 t2

10 km/h

drop-off

Figure B-15 Timing sequence of open loop test

B.6.4.3 Drivers

A professional driver or driving robot shall be used in the subject vehicle.

B.6.4.4 Equipment

Besides the data collection equipment described in Chapter B.1, the following equipment is

necessary during test or test preparations:

- brake temperature sensor

- track surface temperature sensor




The resulting reports from the following inspections are used:






Deliverable D3.2





B.6.4.8.1 Tests

The test procedure addresses μ-split braking according to the following test:

Test 1 μ-split braking (Figure B-16)

The braking manoeuvre consists of braking from a speed of 501 km/h to 0. The steering

wheel is kept at 0˚ during the manoeuvre. The scenario is shown in Figure B-16.

vs

vs

High µ

Low µ

High µ

Low µ

amax

amax

Figure B-16 μ-split braking

The end of the test occurs when the vehicle no longer has two wheels on the low adherence

surface.

During the tests, the parameters of Table B-19 shall be measured.

1 This speed is yet to be finally determined. Please see eVALUE deliverable D4.2 for a discussion on

this subject.

Deliverable D3.2


Table B-19 Measurements


maximal error

distance 0 to 250 m ±1% (≤ 50 m) ±0.50 m (> 50m)

speed 0 km/h to 110 km/h ±0.5 km/h

position 0 m to 250 m X,Y: ± 0.50m

longitudinal acceleration -15 m/s2 to 15 m/s2 ±0.15 m/s2

lateral acceleration -15 m/s2 to 15 m/s2 ±0.15 m/s2

steering wheel angle -360º to 360º ±1 º (<50 º) ±2 º (>50 º and <180 º) ±4 º (>180 º)

yaw rate -50º/s to 50º/s ±0.3 º/s (<20 º/s) ±1 º/s (>20 º/s)

brake force trigger ≤ 10 N (trig point) ±5 N

brake friction material temperature 0-1000° C ±5°C






The stopping distance, which depends on the

longitudinal deceleration, shall be short in a μ-split

braking scenario

mean longitudinal deceleration

The trade-off between stability and stopping distance

is a critical in a μ-split braking scenario

equivalent deceleration

equivalent deceleration on different

tracks

Taking as reference the first 10 km/h speed drop off for the µ-split braking, the following three

safety performance indicators are determined:

Deliverable D3.2


The mean longitudinal deceleration:

2

102

10

1t

t

xF dtaTT

a

The equivalent deceleration:

2

1

2

1110 t

t

x

t

t

X

FED

dta

dtV

aa

The equivalent deceleration on different tracks:

dtV

ttg

aa

t

tX

LOWHIGHLOWHIGH

F 2

112

10 1111

Where:

t1 is the time when driver starts to act on the brake pedal

t2 is the time when vehicle reaches the 10 km/h speed drop off

Vx and ax follow the definitions contained into the glossary, expressed in [m/s] and [m/s2].

Ψ is the yaw rate [rad/sec]

ηHIGH and ηLOW are respectively ratios between the longitudinal decelerations means on high

and low adherence surfaces (obtained from the test described in Test Case paragraph) with

respect to gravity acceleration.

The first safety performance indicator may be the basis to evaluate the longitudinal

performance whereas the second one may be used to evaluate the trade-off between

stability and braking performance.

The last safety performance indicator aims to be a representative parameter of the vehicle

behaviour on µ split surfaces and it should be quite stable for different tracks.

Deliverable D3.2


B.6.4.8.2 Test 1: -split braking

Initial conditions

Drive: gear in neutral position or clutch disengaged (panic brake situation).

Braking manoeuvre

Fast brake application (Fpedal: from 0 to more than 50 daN in 0.15 s)

Brake pedal force is kept higher than 50 daN during the braking manoeuvre.

Steering wheel: the steering wheel is kept at 0 º ( 3º)

Yaw rate: minimum -0.75 º/s, maximum 0.75 º/s

The following applies for the lateral position of the vehicle during the -split braking test case:

the: vehicle longitudinal centreline projection must coincide with the longitudinal borderline

between surfaces. (The wheels on the right side of the vehicle must not enter the left track

surface, and vice versa)

Table B-21 Test cases -split braking

Test case Initial speed [km/h] Friction

1.1 1001 high

1.2 1001 low

1.3 501 split

Each test case shall be performed five times (a valid trial shall fulfil the initial conditions

stated in the respective test). Measures from the trials shall be harmonized (according to ISO

21994). If the test track permits, five additional trials shall be performed in the opposite

direction.

B.6.4.9 Uncertainty

Uncertainty in measurement data shall be stated.


this subject.

Deliverable D3.2


B.6.4.10 Result






Deliverable D3.2


B.7 Emergency braking on a μ-split, closed loop test

B.7.1 Scope


stability control of the subject vehicle when it is exposed to -split conditions. split means

that the left hand side wheels of the subject vehicle is experiencing a significantly other

coefficient of friction ( than the right hand side wheels. The present test procedure

addresses the safety of different types of subject vehicles (car or different types of

commercial vehicles). The test is a closed loop test.

vs

vs

High µ

Low µ

High µ

Low µ

amax

amax

Figure B-17 emergency braking on a -split scenario

B.7.2 References


B.7.3 Definitions


B.7.4 Test procedure -Emergency braking on a μ-split

B.7.4.1 Principle


manoeuvres specified in scenario C3-1: emergency braking on a μ split as stated in Chapter

2.2.

The test aims at determining the performance for the subject vehicle with respect to stability

and stopping distance.


This testing protocol defines tests for closed loop testing where a professional test driver is

used.

Deliverable D3.2


The objective of a closed loop test is to evaluate the overall performance of the vehicle when

considering natural response and feedback from the driver. A driver is used to trigger an

actuation from the vehicle.

Timing sequence of closed loop tests in the emergency braking on a μ split scenario is

presented in Figure B-18.

Start of test

Velocity

stabilized

Braking manoeuvre

started


End of test

t0 t2 t1 t3

Figure B-18 Timing sequence of closed loop test

B.7.4.3 Drivers

A professional driver shall be used in the subject vehicle.

B.7.4.4 Equipment

Besides the data collection equipment described in Chapter B.1, the following equipment is

necessary during test or test preparations:

- brake temperature sensor











Deliverable D3.2





B.7.4.8.1 Tests

The test procedure addresses μ-split braking according to the following test:

Test 1 μ-split braking (Figure B-19)

During the tests, the parameters in Table B-22 shall be measured.


Measure Typical range Recommended maximal

error

distance 0 to 250 m ±1% (≤ 50 m) ±0.50 m (> 50m)

speed 0 km/h to 110 km/h ±0.5 km/h

position 0 m to 250 m X,Y: ± 0.50m

longitudinal acceleration -15 m/s2 to 15 m/s2 ±0.15 m/s2

steering wheel angle -360º to 360º ±1 º (<50 º) ±2 º (>50 º and <180 º) ±4 º (>180 º)

steering wheel torque -30 Nm to 30 Nm ±0.1 Nm (< 10 Nm) ±0.3 Nm (> 10 Nm)

yaw rate -50º/s to 50º/s ±0.3 º/s (<20 º/s) ±1 º/s (>20 º/s)

brake force trigger ≤ 10 N (trig point) ±5 N

brake friction material temperature 0-1000°C ±5 °C

The braking manoeuvre consists of braking from a speed of 100 km/h1 to 0. The driver acts

on the steering wheel to try to make the vehicle run in a straight line. The scenario is shown

in Figure B-19.


this subject.

Deliverable D3.2


vs

vs

High µ

Low µ

High µ

Low µ

amax

amax

Figure B-19 μ-split braking






The stopping distance is shall be short in a μ-split

braking scenario

use of adherence

The stability shall be maintained in a μ-split braking

scenario

stability

The test itself aims at determining the following safety indicators for the subject vehicle: use

of adherence (ε) and stability.

The use of adherence (ε) is calculated as the quotient between the theoretical stopping

distance and the stopping distance using the deceleration at μ-split:

where theorSD is the theoretical stopping distance calculated using the average

deceleration between high and low μ initial braking manoeuvres

µsplitSD is the stopping distance using the deceleration of the µ-split braking

manoeuvre.

The maximum theoretical value ε=100% would represent a very high use of adherence. The

theoretical value ε=0% would represent a very poor use of adherence.

The stability is calculated as:

Deliverable D3.2


high

lowSWRSWAStability

maxmax

where maxSWA is the maximum steering wheel angle achieved during the braking

manoeuvre

maxSWR is the maximum steering wheel rate achieved during the braking

manoeuvre

low is the maximum deceleration achieved on homogeneous braking on low

my surface

high is the maximum deceleration achieved on homogeneous braking on high

my surface

B.7.4.8.2 Test 1: -split braking

An initial test on high surface is performed to determine the stopping distance at high

friction. The stopping distance will then be used to calculate the safety performance indicator

―use of adherence‖.

Initial conditions

Drive: gear in neutral position or clutch disengaged (panic brake situation).

Yaw rate: minimum -0.75 º/s, maximum 0.75 º/s

Steering angle: minimum -3 º, maximum 3 º

Procedure to satisfy the constraint about yaw rate (<0.75 °/s during the 0.5 s before braking

action):

• Maintain a velocity higher than manoeuvre reference velocity

• Disengage the clutch

• Approach to the -split surface

• Control the vehicle (in order to stabilize yaw rate)

• Start braking from the reference velocity

Braking manoeuvre

Fast brake application (Fpedal: from 0 to more than 50 daN in 0.15 s)

Brake pedal force is kept higher than 50 daN during the braking manoeuvre.

Deliverable D3.2


Steering wheel: Driver may act on the steering wheel in order to keep a straight trajectory

and stability. (Driver correction shall be as smooth as possible reducing vehicle’s yaw rate

and without anticipation to vehicle reaction).

The following applies for the lateral position of the vehicle during the -split braking test case.

The vehicle longitudinal centreline projection must coincide with the longitudinal borderline

between surfaces. (The wheels on the right side of the vehicle must not enter the left track

surface, and vice versa)

Table B-24 Test cases -split braking

Test case Initial speed [km/h] Friction

1.1 1001 high

1.2 1001 low

1.3 1001 split

Each test case shall be performed five times (a valid trial shall fulfil the initial conditions

stated in the respective test). Measures from the trials shall be harmonized (according to ISO

21994). If the test track permits, five additional trials shall be performed in the opposite

direction.

B.7.4.9 Uncertainty


B.7.4.10 Result







this subject.

Deliverable D3.2


B.8 Obstacle avoidance

B.8.1 Scope

This testing protocol describes the test procedure for evaluating the safety performance of a

vehicle in an obstacle avoidance manoeuvre. This eVALUE testing protocol requires extra

safety performance indicators to be evaluated during the well-established sine-with-dwell

manoeuvre. However, the manoeuvre itself is performed exactly as described in the ECE

R13-H regulation or in the NHTSA FMVSS126 conformation test.

Figure B-20 Obstacle avoidance scenario

B.8.2 References


B.8.3 Definitions


B.8.4 Test procedure - Obstacle avoidance

B.8.4.1 Principle

The objective of the test is to verify the vehicle behaviour during an obstacle avoidance

manoeuvre. The test is based on the observation of the subject vehicle behaviour when

executing the obstacle avoidance manoeuvre. The test procedure is based on the ESC test

in the ECE R13-H regulation.


This testing protocol defines tests for open loop testing where a steering robot is used.


considering natural response and feedback from the driver. A steering robot is used to trigger

an actuation from the vehicle.

Timing sequence of open loop test in the obstacle avoidance scenario is presented in Figure

B-21.

Deliverable D3.2


Start of test

Velocity

stabilized

Steering manoeuvre

started


End of test

t0 t2 t1 t3


B.8.4.3 Drivers

A steering robot shall be used in the subject vehicle.

B.8.4.4 Equipment

The necessary equipment is specified in the ESC test in the ECE R13-H regulation.













The procedure specified in the ECE R13-H regulation shall be performed.

Besides the measures specified in the ECE R13-H regulation, the following measure shall be

recorded:

Deliverable D3.2


Table B-25 Measurement


error

steering wheel torque -125 Nm to 125 Nm Accuracy 0.3 Nm






Vehicle yaw stability shall be maintained in an

obstacle avoidance scenario

yaw rate ratio at COS+1s and

COS+1.75s

(see ECE R13-H for definition)

Vehicle shall be responsive (lateral displacement) in

an obstacle avoidance scenario

lateral displacement at 1.07s

(see ECE R13-H for definition)

The steering effort required to drive the manoeuvre

shall be within human capabilities in an obstacle

avoidance scenario

1st steering-wheel torque peak

The vehicle yaw response shall follow the driver’s

intentions in an obstacle avoidance scenario

driver intention following

Vehicle roll-over shall be prevented in an obstacle

avoidance scenario

wheel lift

In the ECE R13-H regulation, the following parameters are evaluated: yaw rate ratio and

lateral displacement. These are used to assess the yaw stability and the lateral

responsiveness of the vehicle, respectively. These two are also used as safety performance

indicators in the eVALUE testing protocol.

Besides these two safety performance indicators, three additional are proposed: steering

wheel torque, driver intention following, and wheel lift.

Deliverable D3.2


Steering wheel torque is measured to describe the effort of the driver to perform the

manoeuvre. A very high torque indicates low possibility to control the vehicle. First steering

wheel torque peak is taken.

Driver intention following means how closely the vehicle responds (in terms of yaw motion) to

driver’s intention (commanded by the steering wheel).

time

SW

A

time

yaw

R

time

ay

time

Bra

ke

pre

ss.

time

SW

A

time

yaw

R

time

ay

time

Bra

ke p

ress.

Response gain = Response peak / SWA peak

delay

SWA peak

Response peak

time

SW

A

time

ya

wR

time

ay

time

sl R

EF

time

Bra

ke

pre

ss

time

yaw

R d

iff

time

ay d

iff

time

vx

Response

control

(ay)

Response

control

Response

stability

expected responseactual responseexpected response

actual response

expected

expected - realDIF

0 expected yaw response

> 0 over responsive yaw motion

< 0 under responsive yaw motion

Wheel lift is used to describe roll-over stability. Tip-up criteria is directly carried over from

NHTSA fishhook test. Tip-up condition is met when both inner wheels of the vehicle are

simultaneously lifted more than 50 mm (respect the ground) and for more than 20 ms.

B.8.4.9 Uncertainty


B.8.4.10 Result






Deliverable D3.2


B.9 Highway exit, open loop test

B.9.1 Scope


subject vehicle when exiting a highway at too high speed. The vehicle has to follow a closing

radius trajectory. The present test procedure addresses the safety of different types of

subject vehicles (car or different types of commercial vehicles). The test is an open loop test.

vs

R

Figure B-22 Highway exit scenario

B.9.2 References


B.9.3 Definitions


B.9.4 Test procedure - Highway exit

B.9.4.1 Principle


manoeuvres specified in scenario C3-3: highway exit as stated in Chapter 2.2.

The test aims at determining the performance for the subject vehicle with respect to stability.


This testing protocol defines tests for open loop testing where a steering robot is used.


considering natural response and feedback from the driver. A steering robot is used to trigger

an actuation from the vehicle.

Deliverable D3.2


Timing sequences of open loop tests in the highway exit scenario is presented in Figure B-

23.

Start of test

Velocity

stabilized

Steering manoeuvre

started


End of test

t0 t2 t1 t3


B.9.4.3 Drivers

A steering robot shall be used in the subject vehicle.

B.9.4.4 Equipment

The following equipment is necessary during test or test preparations:













Deliverable D3.2


B.9.4.7.1 Characterization of the lateral dynamics

The Slowly Increasing Steer (SIS) manoeuvre is used to characterize the lateral dynamics of

the subject vehicle. The manoeuvre is used to provide the data necessary for determining

the steering wheel angle (δ0.3g) capable of producing a lateral acceleration of 0.3 g. This

steering wheel angle is then used to determine the magnitude of steering required during the

manoeuvre.

Speed 80 km/h

Ramp steer 13.5 º/s

Figure B-24 Vehicle steer characterisation


B.9.4.8.1 Tests

The test procedure addresses highway exit according to the following test:

Test 1: highway exit (Figure B-22)

During the tests, the parameters in Table B-27 shall be measured.

Deliverable D3.2




error

Distance 0 to 250 m ±1% (≤ 50 m) ±0.50 m (> 50m)

Speed 0 km/h to 110 km/h ±0.5 km/h

Position 0 m to 250 m X,Y: ± 0.50m

Lateral acceleration -15 m/s2 to 15 m/s2 ±0.15 m/s2

Steering wheel angle -360º to 360º ±1 º (<50 º) ±2 º (>50 º and <180 º) ±4 º (>180 º)

Steering wheel torque -30 Nm to 30 Nm ±0.1 Nm (< 10 Nm) ±0.3 Nm (> 10 Nm)

Yaw rate -50º/s to 50º/s ±0.3 º/s (<20 º/s) ±1 º/s (>20 º/s)

centre of gravity sideslip angle

-20° to 20° ±0.3 º/s

The test can be run either as a left turn or a right turn.






Vehicle understeer shall be prevented in an highway

exit scenario

relative radius

Vehicle oversteer shall be prevented in an highway

exit scenario

slip angle

Vehicle roll-over shall be prevented in an highway

exit scenario

wheel lift

The test itself aims at determining the following safety performance indicators for the subject

vehicle: relative radius, slip and wheel lift. The relative radius (Rrel) is the difference between

the trajectory radius in the test run (Ri) and the trajectory radius in the initial test run (R1).

Deliverable D3.2


In all cases, measurement of radius is made at the end of the steering wheel ramp.

The slip angle at the centre of gravity of the vehicle (SlipCOG) is used as an oversteer

indicator.

The wheel lift is used to assess roll stability.

B.9.4.8.2 Test 1 Initial run

An initial test is performed to determine the initial radius.

Speed: 80 km/h.

Drive: highest manual gear or automatic drive in position D.

Curve manoeuvre

The test is performed without throttle (coasting)

Steering rate: 0.3 g/s

Steering angle: 0 to 6.5 * δ0.3g

B.9.4.8.3 Test 2 Successive runs

Successive tests are performed at increasing vehicle speed and steering wheel rate.

Speed: increased by 5 km/h steps from 80 km/h until a final speed of 110 km/h is reached.

Each test case should be performed once.

Drive: highest manual gear or automatic drive in position D.

Curve manoeuvre

The test is performed without throttle (coasting)

Steering rate: increased proportionally to vehicle speed increase (compared to initial run)

Steering angle: 0 to 6.5 * δ0.3g

B.9.4.9 Uncertainty


Deliverable D3.2


B.9.4.10 Result






Deliverable D3.2


Annex C Resolved open issues

This annex presents how the open issues collected in deliverable D3.1 have been treated

and resolved.

Since the closed loop versions of testing protocols in cluster 1 and 2 have been removed to

this version of the testing protocols, the open issues related to those testing protocols are not

discussed in this annex.

Issue: TP:

What is meant with colour range for camera sensors? B2,B3

Resolution:

The colour range of a camera sensor defines the colour spectrum in which light can be detected. It is of course important that the target has a colour within that range.

Issue: TP:

Are all three radii (125, 250, and 500 m) necessary? B2

Resolution:

The different curve radii were inspired by the system classification in ISO 15623. However, in order to reduce the number of trials, 250 m was chosen as the only radius in the test cases of testing protocol B2. (see e.g. B.2.4.8.2)

Issue: TP:

Is maximum 40 ºC test track surface temperature a correctly chosen limit?

B2,B3,B4,B5,B6,B7,B9

Resolution:

The upper bound of track surface temperature was increased to 50 ºC in order to accommodate testing during sunny summer days in southern Europe. (see B.1.4.2)

Issue: TP:

Is 100 Hz sample frequency necessary for video recordings? B2,B3,B4,B5

Resolution:

100 Hz is preferable, but not mandatory. If 100 Hz is used, video data will as frequent as data from high-precision GPS-based measurement systems. (see B.1.5.1)

Deliverable D3.2


Issue: TP:

Tire specification needs to be more specific. B2,B3,B4

Resolution:

A statement ―recommended by the OEM‖ has been added to the tire specification. (see B.1.5.2)

Issue: TP:

Is longitudinal vehicle alignment of ± 0.25 m too tough? B2

Resolution:

The open issue is probably incorrectly formulated since it is the lateral vehicle alignment that shall be smaller than ± 0.25. The issue was raised at a project meeting and it was decided to keep the requirement as it is. (see B.2.4.8.1)

Issue: TP:

The difference between technical indicators and safety indicators needs to be clarified.

B2,B3,B4,B5

Resolution:

Technical indicators have been removed to avoid confusion. (see e.g.Table B-2)

(Technical indicators are related to the technical performance of the system without explicitly telling the implications on safety)

Issue: TP:

The timing sequence of the test puts demands on time gap measurement in real-time, is that feasible? Maybe use clearance instead?

B2

Resolution:

Clearances are used instead of time gaps in the test cases. (see e.g. Table B-3)

Deliverable D3.2


Issue: TP:

Test 1 and 2 in Fig. 3 does not correspond to Test 1 and 2 of the testing protocol.

B2,B3

Resolution:

The figure has been updated to reflect the tests. (see e.g. Figure B-3)

Issue: TP:

―The lateral acceleration of the subject vehicle shall not exceed 2.3 m/s2 hence the maximal speed shall be adapted to the curve radius.‖ Is this a valid requirement? Where does the requirement come from?

B2

Resolution:

The requirement has been removed from the testing protocol. (The origin is probably the ISO 15623 standard which stipulates a maximum lateral acceleration of 2.3 m/s2 for class II and III systems during curve testing)

Issue: TP:

How many trials should there be for each test case? B2,B3,B4,B5,B6,B7,B8,B9

Resolution:

The number of trials for each test case has not been decided. More testing and evaluation of the testing protocols are needed before these numbers can be determined. There is a discussion in D4.2 regarding this issue. (See also footnote in e.g. B.2.4.8.2)

Issue: TP:

Tyre, brakes and system conditioning missing. B2

Resolution:

Tyres and brakes conditioning are described (or rather there is a reference to the conditioning procedure of ISO 21994) in B1.5 in the general part. No proper procedure has yet been defined for system conditioning (the driving style prior to the activation of the system could affect its settings).

Issue: TP:

Which type of lane marking shall be used, worst case? What lane markings represent the worst case?

B4

Resolution:

The eVALUE project has not reached a decision on this issue. Worst case could mean well-painted lines but with low white coverage (dashed lines with long spacing) or it could mean dashed lines with short spacing but with worn lines (little paint left). Lane markings are in this version briefly discussed in the general part, i.e. B.1.4.1 instead of in TP B.4.

Deliverable D3.2


Issue: TP:

―Subject vehicle moves in a straight course at the specified speed, along the centre line of the lane (±5cm) during 100m‖ Is this requirement realistic?

B4

Resolution:

It might be achievable with a steering robot. However the requirement has been removed.

Issue: TP:

Lateral speeds of ≤ 0.3 and ≥ 0.5 are too vague to be used as a test parameter in the test cases.

B4

Resolution:

More precise lateral speed values have been defined in B.4.4.8.2.

Issue: TP:

Is the radius 125 m too small for lane departure tests? B4

Resolution:

Yes. The requirement has been increased to 300-600 m. (See B.4.4.8.3)

Issue: TP:

New directions and targets have been added in the truck specific test cases, is that intentional?

B5

Resolution:

It was intentional since these test cases were based on truck-specific accident data. However it was decided to remove truck-specific test cases from the testing protocol to avoid confusion.

testing and evaluation methods for ict-based safety...

Documents