report d2.1.2 hil2 primary results analysis... project co-funded by the european community within...

www.cats-fp6.aero

Project co-funded by the European Community within the 6th Framework Programme.

REPORT D2.1.2 HIL2 Primary Results Analysis

PROJECT TITLE: CONTRACT-BASED AIR TRANSPORTATION

SYSTEM

PROJECT ACRONYM: CATS

CONTRACT NUMBER: TREN/07/FP6AE/S07.75348/036889

PROJECT START DATE: 01.11.2007

DURATION: 36 MONTHS

PROJECT CO-ORDINATOR: FREQUENTIS AG (1) (FRQ) AT

PRINCIPAL CONTRACTORS: THE EUROPEAN ORGANISATION FOR THE SAFETY OF AIR

NAVIGATION (2) (EEC) BE

AIR FRANCE CONSULTING (3) (AFC) FR

UNIQUE (FLUGHAFEN ZÜRICH AG) (4) (Unique) CH

UNIVERSITY OF LEIDEN, INTERNATIONAL INSTITUTE OF AIR

AND SPACE LAW

(5) (IIASL) NL

EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZÜRICH (6) (ETH) CH

LABORATORIO DI RICERCA OPERATIVA – DIPARTIMENTO DI

ELETTROTECNICA ED INFORMATICA UNIVERSITÀ DEGLI

STUDI DI TRIESTE

(7) (ORTS) IT

ENAV SPA – SOCIETÀ NAZIONALE PER L‟ASSISTENZA AL

VOLO (8) (ENAV) IT

SKYSOFT-ATM S.A. (9) (SkySoft) CH

DOCUMENT IDENTIFIER: D2.1.2

ISSUE: 2.0

ISSUE DATE: 31.05.2010

PREPARED: EUROCONTROL

APPROVED EUROCONTROL

RELEASED FREQUENTIS

DISSEMINATION STATUS: PUBLIC

DOCUMENT REF:

TREN/07/FP6AE/S07.75348/036889 Report number: D2.1.2 Issue: 2.0

File: CATS_D2.1.2_HIL2_Results_v2.0.doc

© CATS Consortium Page: I

History Chart

Issue Date Changed Page (s) Cause of Change Implemented by

0.1 06/01/2010 All sections New document S. Guibert

0.2 08/01/2010 All sections Writing S. Guibert

0.3 15/01/2010 §4, § 5.3.2.3, §6 and §8 updates S. Guibert

0.4 24/01/2010 All sections Finalization J. Gros

1.0 29/01/2010 All sections Final Release C. Rihacek

2.0 24/05/2010 All sections Update after EC comments

J. Gros & S. Guibert

Authorisation

No. Action Name Signature Date

1 Prepared S. Guibert / EEC 24/05/2010

2 Approved J. Gros / AFC 24/05/2010

3 Released C. Rihacek / FRQ 31/05/2010

The information in this document is subject to change without notice.

All rights reserved.

The document is property of the CATS consortium members listed on the front page of

this document. The document is supplied on the express understanding that it is to be

treated as confidential and may not be used or disclosed to others in whole or in part for

any purpose except as expressly authorised in CEC contract number

TREN/07/FP6AE/S07.75348/036889.

The CATS consortium makes no warranty for the information contained in this document,

nor does it assume any legal liability or responsibility for the accuracy, completeness or

usefulness of this information.

The company or product names mentioned in this document may be the trademarks or

registered trademarks of their respective companies.



© CATS Consortium Page: II

Distribution List

This document is distributed as below.

Additional copies held by unnamed recipients will not be updated.

Paper Copy Number Name Address

1 Katarzyna Gryc EC, Brussels

2 Library Frequentis, Vienna

Electronic Copy Number Name Address

1 Katarzyna Gryc EC, Brussels

2-10 CATS consortium



© CATS Consortium Page: III

Contents

1 Introduction .................................................................. 1-1

1.1 CATS Project .................................................................................. 1-1

1.2 Document Structure ........................................................................ 1-2

1.3 Assumptions .................................................................................. 1-3

1.4 Definitions, abbreviations and acronyms ............................................ 1-3

2 Concept of the Contract of Objectives .............................. 2-1

2.1 Concept description ......................................................................... 2-1

2.2 Expected benefits ........................................................................... 2-5

3 Purpose of the experiment .............................................. 3-1

4 Environment of the experiment ....................................... 4-1

4.1 Airspace ........................................................................................ 4-1

4.1.1 Measured Sectors ........................................................................... 4-1

4.1.2 Feed Sectors .................................................................................. 4-2

4.2 Traffic samples ............................................................................... 4-2

4.3 Assumptions .................................................................................. 4-3

4.4 Tools ............................................................................................. 4-4

4.4.1 Controller tools and safety nets ........................................................ 4-4

4.4.2 Pilot tools....................................................................................... 4-7

4.5 Participants .................................................................................... 4-8

4.5.1 Controllers ..................................................................................... 4-8

4.5.2 Pilots ............................................................................................. 4-8

4.6 Training sessions ............................................................................ 4-9

4.7 Schedule ...................................................................................... 4-10

4.8 Data analysis and statistics ............................................................. 4-12

5 HIL2 Results ................................................................. 5-1

5.1 Simulation Facilities ........................................................................ 5-1

5.1.1 Air Traffic Controller platform ........................................................... 5-1



© CATS Consortium Page: IV

5.1.2 Cockpit platform ............................................................................. 5-2

5.2 Human Performance Results ............................................................. 5-2

5.2.1 Controllers' assessment of human performance .................................. 5-2

5.2.1.1 Workload ....................................................................................... 5-2

5.2.1.2 Situation Awareness ....................................................................... 5-17

5.2.1.3 Human Performance ....................................................................... 5-24

5.2.1.4 Working Methods ........................................................................... 5-27

5.2.1.5 Acceptability and Usability .............................................................. 5-44

5.2.2 Pilots' human performance assessment ............................................ 5-44

5.2.2.1 Workload ...................................................................................... 5-44

5.2.2.2 Situation Awareness ....................................................................... 5-46

5.2.2.3 Working methods ........................................................................... 5-48

5.2.2.4 Acceptability and Usability .............................................................. 5-49

5.3 System Performance Results ........................................................... 5-50

5.3.1 Capacity ....................................................................................... 5-50

5.3.2 Safety .......................................................................................... 5-52

5.3.2.1 Separation assurance assessment .................................................... 5-52

5.3.2.2 Post-run Questionnaire and Debriefing Data ...................................... 5-58

5.3.2.3 Safety Conclusion .......................................................................... 5-58

5.3.3 Efficiency ...................................................................................... 5-58

5.3.3.1 Flight Duration .............................................................................. 5-59

5.3.3.2 Number of fulfilled TWs .................................................................. 5-63

5.3.3.3 Efficiency Conclusion ...................................................................... 5-69

5.3.4 Predictability ................................................................................. 5-69

6 Conclusion .................................................................... 6-1

6.1 Human Performances Results ........................................................... 6-1

6.1.1 Controllers ..................................................................................... 6-1

6.1.2 Pilots ............................................................................................. 6-1

6.2 System Performance Results ............................................................ 6-2

6.3 Lessons Learned ............................................................................. 6-2

6.3.1 Training ......................................................................................... 6-2

6.3.2 Platform, HMI and Recordings .......................................................... 6-3

6.3.3 TW Modelling.................................................................................. 6-3

6.3.4 Working Methods ............................................................................ 6-4



© CATS Consortium Page: V

6.4 Future Studies ................................................................................ 6-4

7 References ................................................................... 7-1

8 Annexes ....................................................................... 8-1

8.1 Over The Shoulders Observations: .................................................... 8-1

8.2 Post-Experimental Questionnaire ...................................................... 8-3

8.2.1 Pilots ............................................................................................. 8-3

8.2.2 Controllers .................................................................................... 8-15

8.3 Safety Questionnaire ...................................................................... 8-32

8.4 Situation awareness for SHAPE (SASHA) questionnaire ....................... 8-34

8.4.1 Controllers .................................................................................... 8-34

8.4.2 Pilots: .......................................................................................... 8-35

8.5 Instantaneous Self Assessment (ISA) ............................................... 8-36

Tables

Table 1: HIL2 Validation Plan ............................................................................... 3-2

Table 2: Sectors ................................................................................................. 4-2

Table 3: Feed sectors .......................................................................................... 4-2

Table 4: HIL2 TW values ..................................................................................... 4-4

Table 5: Experiment schedule ............................................................................. 4-11

Table 6: Daily schedule ...................................................................................... 4-11

Table 7: ISA Results for the KL (Geneva) EXE ........................................................ 5-3

Table 8: ISA Results for the MI (Milan) EXE ........................................................... 5-4

Table 9: ISA Results for the KL (Geneva) PLN ........................................................ 5-4

Table 10: ISA Results for the MI (Milan) PLN .......................................................... 5-5

Table 11: ISA Results for the KL (Geneva) EXE with event ....................................... 5-6

Table 12: ISA Results for the KL (Geneva) Planning Controller with event.................. 5-7

Table 13: ISA Results for the MI (Milan) EXE with event .......................................... 5-8

Table 14: ISA Results for the MI (Milan) PLN with event .......................................... 5-9

Table 15: NASA-TLX Results for the KL (Geneva) EXE ............................................ 5-11

Table 16: NASA-TLX Results for the KL (Geneva) PLN ............................................ 5-11



© CATS Consortium Page: VI

Table 17: NASA-TLX Results for the MI (Milan) EXE ............................................... 5-12

Table 18: NASA-TLX Results for the MI (Milan) PLN ............................................... 5-12

Table 19: NASA-TLX Results for the KL (Geneva) EXE with event ............................ 5-13

Table 20: NASA-TLX Results for the KL (Geneva) PLN with event ............................. 5-14

Table 21: NASA-TLX Results for the MI (Milan) EXE with event ................................ 5-15

Table 22: NASA-TLX Results for the MI (Milan) PLN with event ................................ 5-16

Table 23: SASHA-Q Results for the KL (Geneva) EXE ............................................. 5-18

Table 24: SASHA-Q Results for the KL (Geneva) PLN ............................................. 5-19

Table 25: SASHA-Q Results for the MI (Milan) EXE ................................................ 5-19

Table 26: SASHA-Q Results for the MI (Milan) PLN ................................................ 5-20

Table 27: SASHA-Q Results for the KL (Geneva) EXE with event ............................. 5-21

Table 28: SASHA-Q Results for the KL (Geneva) PLN with event .............................. 5-21

Table 29: SASHA-Q Results for the MI (Milan) EXE with event ................................. 5-22

Table 30: SASHA-Q Results for the MI (Milan) PLN with event ................................. 5-22

Table 31: OTS "Overall Performance" results for the KL (Geneva) Sector .................. 5-25

Table 32: OTS "Overall Performance" results for the MI (Milan) Sector ..................... 5-26

Table 33: Total Number of Controllers' Orders for the KL (Geneva) Sector ................ 5-29

Table 34: Total Number of Controllers' Orders for the MI (Milan) Sector ................... 5-30

Table 35: Total Number of Controllers' Orders for the KL (Geneva) Sector with event 5-31

Table 36: Total Number of Controllers' Orders for the MI (Milan) Sector with event ... 5-31

Table 37: Flight Level Orders for the KL Sector ...................................................... 5-33

Table 38: Flight Level Orders for the MI Sector...................................................... 5-34

Table 39: Speed Orders for the KL Sector ............................................................. 5-34

Table 40: Speed Orders for the MI Sector ............................................................. 5-35

Table 41: "Go to" Orders for the KL Sector ........................................................... 5-35

Table 42: "Go to" Orders for the MI Sector ........................................................... 5-36

Table 43: Flight Level Orders for the KL Sector with event ...................................... 5-37

Table 44: Flight Level Orders for the MI Sector with event ...................................... 5-38

Table 45: "Go to" Orders for the KL Sector with event ............................................ 5-38

Table 46: "Go to" Orders for the MI Sector with event ............................................ 5-39

Table 47: Speed Orders for the KL Sector with event ............................................. 5-39

Table 48: Speed Orders for the MI Sector with event ............................................. 5-40

Table 49: Duration of controller communication with piloted aircraft ........................ 5-43

Table 50: NASA-TLX Results for the pilots............................................................. 5-45

Table 51: SASHA-Q Results for the pilots .............................................................. 5-47



© CATS Consortium Page: VII

Table 52: Instantaneous Traffic into the MI Sector along an Experimental Run .......... 5-51

Table 53: Aircraft separation without TW in 2008 traffic conditions for the two controlled

sectors ............................................................................................. 5-53

Table 54: Aircraft separation with TW in 2008 traffic conditions for the two controlled

sectors ............................................................................................. 5-54


sectors ............................................................................................. 5-54


sectors ............................................................................................. 5-55

Table 57: Number of interventions before time to LoS ............................................ 5-56

Table 58: Number of interventions before distance to LoS ...................................... 5-57

Table 59: Flight Duration in KL (Geneva) Sector .................................................... 5-60

Table 60: Flight Duration in MI (Milan) Sector ....................................................... 5-60

Table 61: Flight Duration in KL (Geneva Sector) with Event conditions ..................... 5-61

Table 62: Flight Duration in MI (Milan) Sector with Event conditions ........................ 5-62

Table 63: Percentage of „out TW‟ Aircraft in KL Sector ............................................ 5-64

Table 64: Percentage of „out TW‟ Aircraft in MI Sector ............................................ 5-64

Table 65: Percentage of „out TW‟ Aircraft in KL Sector with event ............................ 5-65

Table 66: Percentage of „out TW‟ Aircraft in MI Sector with event ............................ 5-66

Table 67: Fuel burned Cockpit1 ........................................................................... 5-68

Table 68: Fuel Burned Cockpit 2 .......................................................................... 5-68

Figures

Figure 1: Contract of Objectives ........................................................................... 2-1

Figure 2: Superimposed and adjacent TWs ............................................................ 2-2

Figure 3: TW lifecycle .......................................................................................... 2-3

Figure 4: E-OCVM Lifecycle model ........................................................................ 2-4

Figure 5: Airspace .............................................................................................. 4-1

Figure 6: Picture of the simulation room layout ...................................................... 4-5

Figure 7: „What if‟ dialogue box ............................................................................ 4-7

Figure 8: Cockpit Display ..................................................................................... 4-8

Figure 9: SUPP TW on Navigation Display ............................................................. 5-50



© CATS Consortium Page: VIII

Executive Summary

The main purpose of this document is to present the results of the second Human In The

Loop experiment, conducted as the second step in the operational assessment of the

Contract of Objectives concept.

With regard to the Contract of Objectives (CoO) concept, the controllers had to deliver,

and pilots maintain, aircraft within so-called Target Windows (TWs) - 4D intervals

representing well-defined, agreed and shared objectives for all the actors in the Air

Transport System.

The aim of this specific assessment is to evaluate the operational acceptability of the CoO

and associated TWs concepts from the point of view of the controllers and aircrews.

The evaluation environment addressed two adjacent ACC en-route sectors each

composed of two CWPs managing and coordinating the traffic and two pilots flying two

aircraft.

The assessment focused on:

the human performance of controllers and pilots in terms of workload, situation

awareness, working methods, acceptability and usability,

the system performance in terms of capacity, safety, efficiency and

predictability.

The concept was considered feasible and acceptable by the Air Traffic Controllers

(ATCOs), and the TWs were manageable, even with the 2020 traffic load, without any

impact on safety.

The pilots judged TW management to be feasible and acceptable without any impact on

safety, although the management of the TWs did add certain constraints.

The findings from this second HIL will be very useful for the further analysis of the

concept. Future studies to examine the impact of this CoO in the event of a renegotiation

process will be also conducted in HIL3.



© CATS Consortium Page: 1-1

1 Introduction

1.1 CATS Project

The Contract-based Air Transportation System (CATS) Project was selected during the

4th call of the European Commission FP6. It was begun in early November 2007 for a

period of three years. The CATS Project proposes an integrated decoupling ATM

organisation, where all the actors negotiate and agree on their own objectives. This

organisation is based on shared responsibility between the various actors, whereby the

transfer of responsibility between them is made explicit and formally contracted. This will

reinforce the shared view of the Reference Business Trajectory (RBT) and also the

punctuality of arrivals.

The new Air Traffic Management (ATM) paradigm proposed by CATS is based on the

SESAR concept [6] and proposes one of the possible implementations of the Business

Trajectory, namely the Contract of Objectives (CoO). This proposal introduces an

innovative way of managing ATM by mutually-agreed objectives leading to a market-

driven Air Transportation System (ATS). These objectives represent the commitment of

each actor to deliver a particular aircraft inside temporal and spatial intervals, called

Target Windows (TWs).

It addresses the entire air transport supply chain by reconciling operational links between

air and ground services and thus enhances efficiency by increasing the predictability of

the entire air traffic situation for all ATM actors. Objective assignment and negotiation is

carried out on the basis of the Collaborative Decision-Making (CDM) process, aimed at

establishing the operational agreement (the right balance between productivity and

safety). The context in which aviation is operated is changing rapidly, shifting focus from

capacity growth to flight efficiency, environmental impact and cost reduction. Building a

new ATM system that not only meets current objectives but also supports future ones is

the real challenge taken up by CATS through the negotiation of TWs where all actors'

constraints, current or future, are taken into account, thus supporting business

developments.

A guaranteed performance is offered to the airline by the air traffic management system

to ensure punctuality at destination.

CATS' main concern is to facilitate operational improvements and airspace users'

benefits. The aim of the CATS concept of operations is to improve overall system

efficiency by means of enhanced collaboration focused on the same goal, namely

punctuality at destination.

The objectives presented below reflect the focus of the proposed project:

Link ATM actors together through agreed objectives and interfaces:

Reconciliation of Air Navigation Service Providers (ANSPs), airlines and airports

by ensuring mutual awareness of the constraints (i.e. TWs) imposed by each

actor and thus allowing the ultimate target (i.e. punctuality at destination) to be

focused on. If these constraints are shared, everyone is aware of the possible

options for adaptation in relation to the current flight, thus enabling an efficient

CDM process.




Integrate flexibility to cope with uncertainties: TW modelling respects technical-

level constraints (e.g. aircraft performance, en-route control limitations) and

offers scope for sufficient flexibility when disruptions occur. It gives Air Traffic

Controllers (ATCOs) and aircrew a tool with which to manage uncertainties

arising while a flight is airborne, whilst still respecting the original schedule.

Coordination of actors' resources in order to deliver the best service: The CoO is

the fundamental unit of the collaborative planning process, and is established,

agreed and shared by all the actors. Under normal circumstances, this contract

represents a „guarantee of service results‟ from the actors involved in the flight.

Each actor is thus able to mobilise the relevant resources and infrastructure to

deliver the appropriate service.

Enhanced collaboration through the Single European Sky: The CoO and its

representation though TWs represent a commitment between actors to agreed

interfaces. The air traffic network is highly sensitive to changes (the butterfly

effect), so in order to be efficient it needs to be considered at a regional level

(i.e. on a European scale).

The main aim of the CATS Project is to experimentally assess the operational validity of

the Contract of Objectives proposal.

This will be achieved by:

defining the interfaces (TWs) between various actors and operators:

ATCOs from various ANSPs involved with airborne flight

ATCO and air crew involved with airborne flight

Airport, airline and ANSP during the renegotiation process

assessing, through Human-In-the-Loop experiments, operational acceptability

and technical feasibility, particularly in the event of renegotiation

evaluating the impact of the CoO in terms of human factor issues, safety and

operators' workload

evaluating the benefits (cost business analysis, safety & risk analysis) and the

legal implications.

1.2 Document Structure

This document is the report of the second CATS validation exercise. The purpose of this

document is to record the execution and to present the results and findings of the second

HIL experiment, conducted in Geneva from 19th to 23rd and from 26th to 30th of October

2009.

The main aim of this experiment was to present the CoO concept to controllers and

pilots, obtain feedback on the concept, investigate the impact of the associated TWs on

controllers‟ and pilots‟ current roles and tasks and get an initial assessment of its

operational benefits.

The goal of this second operational assessment was to evaluate the acceptability of this

CoO and associated TWs from a controller and pilot point of view.

This document is divided into 8 main chapters:

Chapter 1 introduces the document




Chapter 2 describes the concept

Chapter 3 explains the purpose of the experiment

Chapter 4 describes the environment of the experiment

Chapter 5 presents the results of the HIL2 experiment

Chapter 6 presents the conclusions of this experiment

Chapter 7 introduces the references used in the document

Chapter 8 - Annexes present the different questionnaires used along the HIL2

experiment

1.3 Assumptions

The CATS project aims to validate the impact of implementing the CoO. This validation

will demonstrate the operational acceptability of TW management and also prove the

benefits and limitations of CoO implementation. A pre-requirement for CATS operational

assessments is that the CoO has already been created, validated and concluded. One of

the assumptions is that all the stakeholders will play the game during the planning

phases to build a stable RBT and CoO. Hence, assuming that the CoO is agreed, this

Human In the Loop (HIL) experiment has been carried out to assess the impact of the

proposed concept on ATCOs and pilots, considering both system and human

performances. The systemic assessment will approach the analysis through a more global

view, including the planning and execution phases.

1.4 Definitions, abbreviations and acronyms

ACC Area Control Centre

ANSP Air Navigation Service Provider

ATC Air Traffic Control

ATCO Air Traffic Controllers

BEC Behavioural and event checklist

CAP Capacity

CONOPS Concept of Operations

CoO Contract of Objectives

CWP Control Working Position

DST Dynamic Scanning Tool

ECAC European Civil Aviation Conference

EFF Efficiency

E-OCVM European Operational Concept Validation Methodology

EXE Executive Controller

FAA Federal Aviation Authority

FL Flight Level




FMS Flight Management System

HF Human Factors

HFE Human Factors Expert

HIL Human In the Loop

HMI Human Machine Interface

HST Horizontal Scanning Tool

ISA Instantaneous Self Assessment

KPA Key Performance Area

KPI Key Performance Indicator

MCDU Multipurpose Control and Display Unit

MTCD Medium Term Conflict Detection

NASA-TLX NASA Task Load indeX

ND Navigation Display

NOP Network Operational Plan

OTS Over The Shoulder method developed by FAA

PF Pilot Flying

PLN Planner controller

PNF Pilot Non Flying

PI Performance Indicators

PRED Predictability

RBT Reference Business Trajectory

SA Situation Awareness

SASHA Shape Questionnaire

SESAR Single European Sky Applied Research

SME Subjects Matter Expert

STCA Short-Term Conflict Detection

SWIM System Wide Information Management

SWOT Strengths, Weaknesses, Opportunities and Threats

TWs Target Windows

SA Situation Awareness

SAF Safety




2 Concept of the Contract of Objectives

2.1 Concept description

A new way to enhance traffic efficiency is to improve functional and operational

continuity in aircraft management, both on the ground and in the air, with a view to

meeting safety and productivity objectives. Functional continuity has an airspace

dimension (heterogeneity of European airspace) and a time dimension (from planning to

execution). To this end, it is proposed to bring together all air navigation components by

means of a Contract of Objectives.

The purpose of the CoO is to create an operational link between all air navigation actors

(airlines, airports and ANSPs). The CoO represents a formal and collaborative

commitment between all the actors in the ATS. It establishes the role as well as the tasks

and responsibilities of each party, based on well-defined, agreed and shared objectives.

These objectives represent the commitment of each actor to deliver a particular aircraft

inside temporal and spatial intervals, called Target Windows (TWs). These commitments

are agreed by all involved actors for specific transfer of responsibility areas (e.g. between

2 ACCs). Then, each actor will be fully accountable for its own achievements. The

ultimate objective of the CoO is punctuality at the destination, while improving the

system efficiency and predictability by means of enhanced collaboration between air

transport actors.

Contract of Objectives

ApproachANSP1Airport ANSP2 AirportTWR TWR

On groundOn ground On FlightTaxiing Taxiing

Off-BlockTime

LandingTakeoff

Control Unit

Control Unit

Control Unit

Control Unit

Control Unit

Control Unit

Target Windows

In-BlockTime

Air side

main objective

Ground side

main objective

1 Flight

Figure 1: Contract of Objectives

In order to formalise the Contract of Objectives and its refinement for each local actor, a

concrete manifestation of the CoO is proposed through the TWs. TWs create a common

object between all the operators involved, and also between the planning and operational

phases. Instead of precise 4D points, the TW is expressed in terms of temporal and

spatial intervals. They are defined on the basis of transfer of responsibility areas (Figure

1). Their sizes and locations reflect negotiated objectives resulting from downstream




constraints, such as punctuality at the destination, runway capacity, congested en-route

areas or aircraft performance, airline operations. TWs provide room for manoeuvre to

ensure resilience in case of disruption and conflict management and, lastly, impose

constraints only if necessary. Uncertainty will always be a component of the system and

can never be entirely erased. The CATS concept proposes, instead of removing this

uncertainty, to keep it under control by managing disruption via the size of the TWs and

to limit the side effects of any disruption. Divergence from schedule (either due to

operational issues or owing to uncertainty) still remains possible, but if so it triggers a

specific decision-making process, called renegotiation, at a system-wide level.

Figure 2: Superimposed and adjacent TWs

These TWs are negotiated, utilizing a collaborative decision-making (CDM) process

supported by system-wide information management (SWIM), in terms of punctuality at

the destination, taking into account all actors' constraints. This negotiation process can

be described as follows:

long-term planning phase (from years to months): development of an initial

schedule, not overly detailed, constituted by TWs at departure and arrival

airports, taking into account infrastructural and environmental constraints;

medium-term planning phase (from months to days): development of business

trajectories and negotiation of TWs through an iterative process; integration of

weather predictions;

short-term planning phase (from days to hours before the execution phase):

continuous refinement of the TWs up to CoO signature.




TWs

Airlines/Airports

SWIMNET

TW

negotiation

TW

refinement

CoO

TW signed

LIFECYCLE OF

TARGET

WINDOWS

(Refinement)

TWs

Airlines/Airports

and ANSPs

RENEGOTIATE

Figure 3: TW lifecycle

Then, the execution phase of the flight can start. The CoO gives the controller and

aircrew a means of managing the uncertainty inherent in air traffic in accordance with

their own objectives. The crews' objectives, therefore, are to adhere to an arrival

schedule defined through TWs. Controllers, on the other hand, must ensure aircraft

safety while keeping aircraft within the TW defined in the contract, which guarantees that

the contract will be honoured.

For this purpose two different views of these TW objectives will be implemented in the

Controller Working Position (CWP):

A global representation, needed for ATCOs to monitor whether the flight is in or

out of the TW. This data will be available on the label and will indicate the status

of the aircraft compared to the TW:

IN

OUT

EDGE/BORDER

A detailed view of each dimension, useful for conflict resolution or any controller

actions. A support tool, along the lines of “What if” will also be implemented.

If, for any reason (weather changes, etc.), one of the TWs cannot be fulfilled, a

renegotiation process will start between the impacted actors, resulting in a new CoO.

Currently, responsibility for fulfilling the CoO remains with the airspace user, in order to

be consistent with the current regulation whereby the airline is strictly liable and also

because the airline has a more global view of the overall flight. This point will be

examined later in the project, mainly in WP2.1.3 and WP2.2.3.




The renegotiation process is performed with the actors using SWIM network facilities. The

corresponding communication services are optimized (the amount of data exchanged

minimized) to avoid saturation of the SWIM network. A revision, involving the proposed

change to a Target Window, may be proposed by the ANSP, airport, airline or aircrew.

Several important principles are applicable here:

When the time horizon allows, the revision of the TW should use a CDM process

involving all the actors concerned, but mainly the airspace user, to ensure the

best possible business outcome;

In certain cases, e.g. if a TW renegotiation involves only two ANSPs, the process

is simplified (point-to-point). The outcome of the TW renegotiation process is

then made available using the SWIM network;

When the situation is urgent, the controllers may decide to immediately and

locally revise the trajectory for safety and separation purposes, without applying

a CDM process.

The SESAR CONOPS [7] changes the approach of ATM to a performance-based approach.

Trajectory-based operations ensure that the actual trajectory flown by the airspace user

is close to its intended one, integrating ATM and airport constraints. The proposed CoO

consists of a collection of TWs between each area, where responsibility between actors is

transferred. The Business Trajectory (proposed by SESAR) should then go through these

different TWs, in order to ensure the system‟s predictability (compliance between what is

planned and what is flown) and overall efficiency.

According To E-OCVM [9], the level of maturity of the CATS concept extends from V0 up

to V2 (Figure 4).

V1

Scope

V2

Feasibility

V3

Integration

V4

Pre-operation

V 5

Operation

V0

ATM Needs

Idea

Implemented

Concept

Identify ATM

performance

needs &

constraints

Scope operational

concepts and create

validation strategy

Iteratively

develop and

evaluate

concept

Integrate concept in

wider context

And confirm

performance

Industrialisation

and procedure

approval

Implementation

E-OCVM Concept Lifecycle Model

CATS

Figure 4: E-OCVM Lifecycle model




The CATS project will cover the following steps:

V0: ATM Needs: The aim of this step is to identify the ATM performance needs,

and current constraints/barriers. Most of this task has been developed by SESAR

D2 [6] and will be the basis of our work. The CATS D1.1 results also added

interesting research questions.

V1: Scope: The goal there is to describe the proposed solution and highlight the

benefits foreseen. This will be the role of WP1, through Deliverable D1.2.1. This

step also covers the definition of the Validation Strategy.

V2: Feasibility: The aim of this task is the evaluation of the proposed concept. It

is an iterative process to show the operational feasibility of the proposed

solution. Within the CATS project, this task is foreseen during the different

assessments proposed in WP2 [2].

As presented above, the CATS project investigates solutions to the "arriving on schedule"

concept while improving efficiency and synergy between all ATM actors through Contract

of Objectives. These foundations are fully in line with the SESAR concept, to face the

challenge of traffic growth (2020+ horizons). The objective of CATS is to assess the

feasibility of such a proposed concept and not to introduce it in the current operations

within the next few months. The first experiment described here was just a proof of

concept study. The CATS project is at the beginning of the concept development and

then, at an early stage in the E-OCVM level of maturity (V1 and V2 level, as shown in

Figure 4).

2.2 Expected benefits

At conceptual level, the CoO and TWs can be regarded as an operational way of achieving

the establishment of the ATM performance partnership recommended by SESAR [6]. TWs

represent a possible means by which all the stakeholders can share a unique and

impartial view of each other's priorities. Thus, they ensure a common translation and

representation of the performance targets to be achieved by the overall ATM chain.

At a second more operational level, TWs unequivocally identify the transfer of

responsibility areas between partners, and at the same time they constitute a way of

managing uncertainty and monitoring disruptions. Measurement of compliance with TWs

established during the negotiation process could represent a new and reliable metric for

assessing the quality of a service provided.

The CoO and TW concepts are expected to directly bring the following substantial

benefits to the ATM system:

More punctuality at the destination (arrival-on-time concept): the CoO concept

proposed in the CATS Project is designed to achieve an ultimate goal, namely

arrival on time at the destination airport. Through the CoO, aircrew, controllers

and airports share the same goal for the flight represented by an agreed

contract. The synergy between the air and ground components is thus

reinforced. Airlines will reduce delay-related costs and optimise their aircraft

turn-around times. Airports will be able to optimise their ground operations.

Even though the efficiency design target identified by SESAR applies to on-time

departure, there is clearly a strong correlation between punctuality at departure

and at the destination. It would be interesting to evaluate this correlation during

the assessment process.




Optimisation of scarce resources: during the design/drafting of the contract,

through the Network Operational Plan (NOP), the actors‟ constraints will be

taken into account in the collaborative process. Airlines will indicate their

economic and technical constraints (i.e. business trajectories) in the negotiation.

This will allow airlines and other actors to respond appropriately to the initial

demand, which is in line with their constraints. Airports will be able to optimise

runway use (through better scheduling) and thus improve throughput.

Furthermore, their constraints will be integrated at an early stage of the

collaborative process. ANSPs will be able to optimise their resources, since they

will be responsible both for their local airspace design and for working methods

in fulfilling contracts previously agreed with other actors. Furthermore, during

the drafting process for the CoO, they will be involved at an early stage and thus

be able to indicate their constraints in the trade-off mechanism. This

optimisation of resources will bring benefits in the key performance areas of

cost-effectiveness and efficiency, since the enhanced allocation of scarce

resources among actors will positively impact the efficiency of the entire air

transport supply chain.

Improved predictability: the TWs are designed taking into account aircraft

technical constraints, with built-in scope for disruption management in order to

achieve the ultimate target of the CoO, which is "arrival on time at the

destination". Each actor knows its part of the contract, i.e. those TWs it must

fulfil. Airlines will be able to rely on their schedules, as predictability will be

improved, and they should get a better pay-off from their fleet. Airports will also

be able to rely on their schedules, and so optimisation of ground operations will

be possible. This will not only enhance the quality of service delivered to users

(both airlines and passengers) but also improve the infrastructure pay-off.

ANSPs will have ensured consistent airspace design and provide the necessary

manpower in line with the expected level of traffic. Controllers will be able to

better anticipate the traffic by having a global view of the system (the TW

defines the constraints for punctuality at the destination). In line with SESAR

requirements, variability of flight duration will be kept to a minimum, and service

disruptions will be promptly managed and solved by the actors involved through

the renegotiation process.

Reduced overall costs: this aspect is closely linked to previously mentioned

benefits, as optimisation of resources and improved predictability naturally lead

to reduced costs. Airlines will be able to place more trust in scheduling. This will

allow them to improve turn-around patterns, and thus improve their response to

passenger demand. Airlines will be able to fly as close as possible to their

business trajectories, and will then benefit from a trajectory-based organisation.

Airports will get a better approach and better scheduling of their ground

operations, and will thus be able to dedicate the right number of resources to

service provision, which in turn will lead to cost-efficiency. ANSPs will be able to

better anticipate airspace opening arrangements and design as well as

manpower needs, which will allow them to adjust the size of their teams so as to

improve efficiency. The cost-effectiveness of the system deserves detailed

investigation to ensure that cost improvement is achieved via this concept.

The CATS Project represents one possible solution to another issue highlighted in SESAR

D2 [6], namely the need to determine "how to deal with business trajectories in the

strategic, tactical and operational phases of flight", since the CoO is a possible way of




implementing the business trajectory, the notion around which the future ATM system

will be designed.

Through Network Managers, ANSPs, Airports and Airlines working together to define and

agree an optimum trajectory that is safe, efficient, economic and acceptable to the

environment, the CATS project could provide the ATM community with an opportunity to

further understand the SESAR Business Trajectory and how it may operate. At least it will

input significant understanding to the validation required for complex concepts.




3 Purpose of the experiment

The main aim of the CATS Project is to assess the CoO and associated TWs by involving

the major actors in the supply chain (i.e. airlines, airports, and ANSPs).

The aim of this specific assessment is to evaluate the operational acceptability of the CoO

and associated TW concepts from the controllers' and aircrews‟ point of view.

The objective of the HIL2 experiment is twofold:

Analyze the collaboration process between controllers and aircrew regarding the

TW management

Analyze the impact of the TW management on the aircrew's activity in the

cockpit.

This will be evaluated in the context of the transfer of responsibility between two ANSPs.

The evaluation environment is restricted to two en-route controller working positions

(CWPs) managing the traffic and coordinating the aircraft (i.e. the transfer mechanism).

During the experiment some aircraft will be piloted by real aircrew, on a Flight Simulator

position, and the other aircraft will be managed by the simulation platform itself.

The hypotheses to validate through this assessment are:

CoO implementation allows safe operations.

CoO is still manageable even with the traffic growth foreseen to 2020.

CoO implementation positively affects the flight within the sector (flight duration,

etc.).

Implementation of TWs ensures that the schedule is respected.

TWs integrate flexibility to cope with uncertainty.

The working methods offered to ATCOs and pilots as a result of CoO

implementation are feasible and acceptable (task-sharing, role and

responsibility, as well as the support tools offered).

Implementation of CoO does not impose significant additional workload on

ATCOs or pilots.

The idea is first to analyse how the proposed CoO and the associated TWs will impact

system performance regarding some selected KPAs, linked with SESAR expectations. The

contribution of humans to overall system performance will therefore also be analyzed.

Table 1 below represents the different KPAs and KPIs which will be used during this

experiment to assess system performances, and how human performance will interact.




HIL 2 Assess the impact of CoO between ATCOs and

aircrew

Objectives relating to system

performances (KPA)

Objectives relating to human

performances

SAF G1.1 & G1.2: Feasibility and acceptability of the aircrew and ATCO working methods as a result

of CoO execution

G2.1 & G2.2: Impact of CoO execution on aircrew and ATCO performance

G3.1 & G3.2: Impact of CoO on aircrew and ATCO activity

CAP

EFF

PRED

Indicators (KPI) Indicators

SAF.LOCAL.ER. PI (1, 2, 3, 5, 6 & 8) Workload: ISA, NASA-TLX,

Interviews, Observations, Performance outcomes,

Questionnaire

Situation Awareness: SASHA_Q,

Interviews, Observations, Performance outcomes, Questionnaire

Error production and management: Observations, Questionnaire, Interviews,

Performance outcomes

Operator's activity: Cognitive processes, Decision making, Risk management, Constraints, etc.

Collaborative and R/T activity: Communications (number, time, content, speaker and receiver, etc.)

CAP.LOCAL.ER. PI (2, 8, 10, 11, 12 & 13)

EFF.LOCAL.ER. PI (1, 7, 8, 9, 10, 11)

PRED.LOCAL.ER. PI (1 & 2)

Number of TWs fulfilled

Table 1: HIL2 Validation Plan

From the catalogue of Performance Indicators, delivered by Episode 3 project [10], CATS

extracted some Local Performances Indicators relating to its concerns and also added

some indicators specific to the concept. As CATS Operational assessments evaluate the

operational acceptability of the CoO and associated TWs concepts from the actors' point

of view between two ACCs, and not the ECAC-wide performance of En-Route, we only

focus on the Local Performance Indicators (PI) layer.

In order to meet the objectives of Table 1, two independent variables, as defined in

D1.3.4 (Scenarios) [17] will be used during the HIL2 experiment:




Usage of the Contract of Objectives and Target Windows: each scenario will be

assessed with and without Target Windows, to measure the impact of the

introduction of TWs.

The traffic load: two levels of traffic load will be evaluated, the current one

(2008) and the one expected in 2020, according to EUROCONTROL forecasts.

During the scenario, weather disruptions (cumulonimbus and wind) will also

occur and will be referred to as "events" in the rest of the document.




4 Environment of the experiment

The following information is set out in detail in the D1.3.4, CATS HIL2 Experimental

Scenario Description [17].

4.1 Airspace

The airspace chosen for this first experiment is the same as that used for the FASTI real-

time simulation.

The controlled sectors are located in the Europe Core Area, during busy traffic hours.

As this simulation is looking specifically at the En-Route environment, the simulated

airspace involves two ACCs, as described in the concept [4], mainly cross-border

operations between the Swiss and Italian airspace.

4.1.1 Measured Sectors

Two measured sectors, located at the frontier between Rome and Geneva ACCs have

been chosen for this first experiment.

The simulation airspace is based on the border area between Geneva and Rome, mainly

the KL1 and MI1 sectors, with appropriate feed sectors, as presented below.

Figure 5: Airspace

KL1

FL275 – FL345

MI1

FL275 – FL345




ACC Sector name Min FL Max FL

LSAG LSAGKL1 FL275 FL345

LIRR LIRRMI1 FL275 FL345

Table 2: Sectors

Both in Geneva and in Rome smaller sectors have been grouped in order to obtain larger

sectors and hence facilitate TW adherence.

The route network is the same as the current one. Measured sectors are managed by one

executive and one PLN. Standard separations are applied during the experiment.

4.1.2 Feed Sectors

The feed sectors have been developed to ensure the continuity of control and

coordination with the measured sectors. The control actions of these positions have been

kept to a minimum. All feed sectors are hybrid positions, without any human operator. All

aircraft manoeuvres are performed automatically, following the traffic prepared.

Therefore, human support is not required.

The following feed sectors were simulated:

Sector name Min FL Max FL

Feed East FDE 0 UNL

Feed West FDW 0 UNL

Feed Geneva Inf. KL2 0 FL275

Feed Geneva Sup. KL3 F345 UNL

Feed Milan Inf. MI2 0 FL275

Feed Milan Sup. MI3 F345 UNL

Table 3: Feed sectors

4.2 Traffic samples

The traffic samples used in the human-in-the-loop simulation derive from the traffic

samples used in the FASTI RTS B simulations. Real traffic in July 2008 has been recorded

and modified to meet the SESAR 2020 objectives (real traffic samples, increased by

STATFOR percentage in this specific area).

Each scenario lasts one hour and encompasses two traffic levels: in 2008 and in 2020.

This means the first part of the scenario will simulate the 2008 traffic and the second

part will simulate the forecasted 2020 traffic.

The reasons for which the order of the two traffic levels is always the same are:

Providing a warm-up period to the ATCOs.

Analysing the transition from the low level to the high level of traffic.

Identifying any „safety‟ or „efficiency‟ breaking point, if such a point arises when

the traffic load increases.




The script is always the same for all scenarios and encompasses the following phases:

The "Launch" phase. This phase is the initial phase in which the scenario starts

and the first aircraft are simulated. The "launching" phase ends when the 2008

traffic level is reached. This phase lasts approximately 10 minutes.

"2008 traffic load" phase. This phase lasts approximately 20 minutes. Several

conflicts are planned in this phase to test the ATCOs' ability to manage conflicts

in the CoO frame.

"Transition" phase. This phase corresponds to the change of traffic load level

between 2008 and 2020. This phase lasts approximately 5 minutes. Conflicts

occur during this phase.

"2020 traffic load" phase. This phase lasts approximately 20 minutes. As for

2008 traffic load, some conflicts are planned.

"Disruption" phase. This phase spans the last 10 minutes of the scenario. A

disruption, due to activation of a restricted area, or sudden weather changes,

occurs within the scenario.

"Final" phase. The "final" phase will occur when the traffic load begins to

decrease. The end of the scenario run is decided by the experimenter depending

on how the ATCOs manage the traffic and the loss of interest for the experiment

in pursuing the run.

4.3 Assumptions

The airspace is being managed and the controlled sector is located in the Core Area,

during busy traffic hours. The information known to one actor will be disseminated

through the System-Wide Information Management (SWIM) and is considered to be

available to all parties concerned.

Additional information and assumptions are the following:

The CoO and associated TWs have been negotiated and shared by all the actors

involved. All flights within the airspace have entry and exit TWs.

All actors concerned are aware of the contents of the Network Operations Plan

(NOP) and of the TWs.

The renegotiation process is out of the scope of this experiment. If the ATCO

cannot manage exit TW for a flight, this flight will be transferred without any

specific coordination. For the purposes of the experiment, the number of TWs

not achieved will be counted as an indicator.

The crew is responsible for achievement of the CoO, although ATCOs should also

do their best to adhere to TWs.

Each measured sector is managed by one ATCO pair: executive and planner.

Both of them perform some tasks on a Controller Working Position (CWP).

Each pilot position is managed only by one pilot. Communications between crew

and ATCOs are supported by radio telephony.




Only 4 aircraft1 per run are piloted by real aircrew. The others are managed by

automated pseudo pilots. Then, the control orders are given by the ATCOs to

aircraft through the radar screen, via pseudo data link messages. These orders

are automatically accepted by the aircraft, but the system will introduce variable

delays.

All flights entering the controlled sector are inside their entry TW. For HIL2, as

the TW model calculation is still ongoing (WP2.2.4), the TW values taken into

account for this experiment, extracted from an in-between model, were:

TW width

(NM)

TW time width

(min)

TW level width

(00 feet)

Average 14.45 7 44

Min 8.64 5 0

Max 21.37 17 60

Std. Dev. 2.89 2 14

Table 4: HIL2 TW values

4.4 Tools

4.4.1 Controller tools and safety nets

The platform used for this experiment was the SkyGuide simulator, adapted by SkySoft-

ATM.

1 In D1.3.4, five aircraft were prepared per run: 3 short haul flights and 2 long haul flights.

Unfortunately, after the validation session, the sequence of three aircraft was not manageable. It was decided to have only 4 piloted aircraft per run.




Figure 6: Picture of the simulation room layout

The standard Geneva services and tools were included in the platform, such as:

HST (MTCD1): The purpose of the Horizontal Scanning Tool (HST) is to detect

conflicts between aircraft, based on their current clearances, in the horizontal

plan. The HST runs continually on each track update, identifying flights with

conflicting Current Flight Level (CFL) (RVSM flights: the same CFL +/- 990 feet;

non RVSM flights: the same CFL +/- 1990 feet), displayed when the minimum

horizontal separation determined from their trajectories falls below 10NM or

15NM (depending on personal set-up). The user can set to display/hide the

following elements for tracks involved in an HST conflict:

the conflict vector

the speed vector

the full route or the route until the conflict

The user can also acknowledge the conflict or change the minimum separation

distance limit to filter the HST conflicts in the HST window.

DST (MTCD2): The purpose of the Dynamic Scanning Tool (DST) is to assist the

controller in detecting possible conflicts that would result from a given CFL,

heading or direct clearance being issued to an aircraft. In this case, a display

allowing for filtering on sector height and more generally on previously accepted

conflicts is applied to efficiently dispatch the conflicts to the controllers

concerned. The DST conflicts are show in a window displaying medium-term

conflicts provided by the SkyServer. When a DST conflict is detected the user

can display the following elements:




the conflict vector

the speed vector

the tendency

the remaining time

the CFL of the track

The user can also choose to display the DST conflict for the next 5, 7, 10 or 15

minutes. He can choose to display the DST window at an offline position or near

the track label that has generated the DST conflict.

Only one DST calculation can be performed at a time, otherwise a warning

message is displayed.

The DST conflicts can be acknowledged or cancelled by the user.

AMT (MONA): The Adherence Monitoring Tool (AMT) is monitoring different

clearances given to pilots entered through the label in the radar visualization

system: e.g. heading, direct routes or alternatives routes (graphical route

modification). The Adherence Monitoring Tool is used to monitor the flight path

according to the route entered by the controller. In case of deviation of the

track, an alert is displayed on the label with the corresponding field coloured

orange. Due to the pseudo-pilot automation tool, usage of the AMT is very

limited. It makes sense only if a track is modified manually through the pseudo-

pilot HMI.

Silent coordination (SYSCO): The objective is inter-sector silent coordination

from screen to screen: quicker, optimises working time, more explicit than the

phone, possibility of immediate multi-sector co-ordination, useful for sector

altitude layer configuration with evolutionary traffic. To initiate a coordination,

the controller simply has to enter a coordination value (i.e. suggested flight level

and corresponding climb/descent rate, heading, DCT route or speed) through a

Coordination Window, and send it to a receiving sector, usually an adjacent

sector (or several layers of adjacent sectors) which has the flight in contact. The

flight is intended to enter the airspace volume of the transferring sector. Silent

coordination is used only for the exercises without TWs. The TW is the

coordination tool when TWs are used.

STCA (Short Term Conflict Alert): Short term conflict alert (STCA) designed to

warn the controller of any situation where a pair of radar tracks for which

minimum separation distances are, or are predicted to be, violated in a short

look-ahead time (2 minutes).

Specific HMI for TW display and tool such as „What if‟ have also been designed by

SkySoft-ATM [18] for the purposes of the CATS experiments:

The „what-if‟ HMI: „What if‟ is a tool implemented as a dialog box that allows an

ATCO to probe, before validation, some modification of route, level and/or speed

for an aircraft. With the help of sliders, the ATCO sees immediately the impacts

of these modifications on the TW. He also sees the impact on the trajectory as it

is displayed automatically. This tool is activated by a mouse click on one of the

TW status field displayed in the extended label. This support is designed for an

anticipative approach. It is an assessment of a trajectory update on a single

aircraft basis.




The What if tool in the HIL2 becomes a support for coordination between

PLN/EXE. For example, when the planner prepares a solution the label on the

screen remains in the current state, while the proposal appears as a dotted line,

along with the result regarding the TW. The planner could then propose this

solution to the EXE, who could accept or refuse it, through the „what if‟ entry list.

If the solution is accepted, the planner should then validate it.

Figure 7: „What if‟ dialogue box

Apart from the two piloted flights, there was no pseudo pilot and feeder sectors were

unmanned. This led to:

Automatic execution of orders by aircraft

Automatic handover/acceptance by feeder sectors

However, in order to avoid decreasing the ATCO workload too much, a data link device

was incorporated into the ATCO working position. The data link will integrate latency

delays between the orders, their acknowledgement and their execution by aircraft. The

latency delays will vary randomly between 10 seconds and 30 seconds.

4.4.2 Pilot tools

Two captain cockpit panels were used for this experiment. Skysoft ATM used Flight

Simulator 2004 with two add-ons:

one for the Airbus A320 (simulating the panels and performance of the aircraft),




one for the FMS and the MCDU of A320.

Figure 8: Cockpit Display

4.5 Participants

4.5.1 Controllers

Four controllers from ENAV participated in the simulation:

Two from Rome ACC

Two from Brindisi ACC

All of them had been qualified for over 10 years and are currently working as controllers

in en-route sectors. The instructions given by the validation team to the ATCOs were to

control traffic by ensuring safety and respecting the exit TWs.

4.5.2 Pilots

Two pilots recently retired from Air France (less than 4 months ago) participated in the

simulation:




One has 32 years' experience as a pilot (17,000 flight hours) with 19 years as

captain on A320, A330/340 and B747-400.

The other has 35 years' experience as a pilot (19,600 flight hours) with 20 years

as captain on B737, A330/340 and B747-400.

The instructions given by the validation team to the pilots were to ensure the

safety of the flights and to respect the TWs.

4.6 Training sessions

As with all simulations, the importance of a training program is essential to fully evaluate

the proposed concept.

The aim of this training program was to:

present the concept and experiment to the controllers and to the pilots,

familiarize the controllers with the HMI, working methods and airspace,

familiarize the pilots with the simulation cockpit, and

introduce the data collection and analysis techniques/tools used during the

experiment.

The training program first started with a one-day presentation meeting held in Paris

three weeks before the experiment period. This presentation consisted of oral and Power

Point presentations of the concept, the objectives of the experiment and the environment

and devices used. The purpose was to make sure that the pilots had fully understood the

expectations of the experimenters. This meeting was also the occasion to dynamically

familiarize the pilots with the simulation cockpit.

There was a one-day presentation meeting held in Rome one week before the experiment

period. This presentation consisted of oral and Power Point presentations of the concept,

the objectives of the experiment and the environment and ATCO devices used. The

purpose was to be sure that the ATCOs had fully understood the expectations of the

experimenters. This meeting was also the occasion to dynamically familiarize the ATCOs

with the HMI.

Besides the training sessions held in Paris and Rome, a training session in situ was also

planned during the experiment period. This training started the experiment, after a short

presentation, as follows:

One day for familiarization with the simulation devices, HMI, airspace, tools and

CoO, simulated cockpits. The aim of this training was pedagogical. It was

interspersed with explanations, comments and even on-line adjustments of the

working methods. It was therefore possible to stop/freeze the simulation.

One day and a half, for training purposes, on operational scenarios and the

experimental environment. The training was run in the same conditions as the

measured simulation (training scenarios with traffic loads and events equivalent

to those in the measured simulation). These training sessions allowed

experimenters to stop/freeze the experiment, in order to give explanations and

make final changes and adjustments.




At the end of the training session, a debriefing was planned to evaluate the

controllers‟ and pilots' feedback.

4.7 Schedule

The experiment took place over ten days at the Skyguide premises in Geneva-

Switzerland:

Five days from 19-23 October 2009.

Five days from 26-30. October 2009.

The experiment period timetable encompassed:

Half a day to present the simulation devices: functions and limits.

One day for familiarization

One day and a half for training

Six days for performing the 16 experimental runs with a maximum of 3 runs per

day.

One spare day to cope with unexpected simulation device events (i.e.

breakdown, etc.), not used during this experiment.

Final debriefing with all attendees closing the HIL2 experiment period.

Table 5 below presents the schedule of the ten days of experiment.




Day Morning Afternoon

19. October Simulation device presentation Familiarization

20. October Familiarization Operational training (session 1)

21. October Operational training (session 2) Operational training (session 3)

22. October Experimental runs #1, 2 & 3


26. October Familiarization and Training Experimental runs #7 & 8



29. October Experimental runs #15 & 16 – Final Debriefing

30. October Spare day

Table 5: Experiment schedule

Simulation exercises were conducted on the basis of three exercises per day, each

running for about 1 hour, plus 30 minutes for filling-out questionnaires.

Each run encompassed:

A short run presentation and briefing.

The actual performance of the run (70 minutes).

The completion of questionnaires and self-evaluation scales (15 minutes)

A debriefing (20 minutes)

A typical daily schedule was as follow:

Daily Programme

TBD Depart Hotel SkySoft arrival 0830

0845 Set-up in Operations Room

0900 – 1045 Exercise 1+Debriefing & Questionnaires

1045 – 1100 Break


1245 – 1415 Lunch


1615 Leave

Table 6: Daily schedule




4.8 Data analysis and statistics

System and human performance objective data were automatically recorded and

processed by statistical methods.

Questionnaires and self-evaluation scale data were processed by statistical methods. A

non-parametric test, Wilcoxon-Mann-Whitney, was chosen for assessing whether two

samples of observations come from the same distribution. It is one of the best-known

non-parametric significance tests.

Other subjective data gathered via observations and verbal comments were processed by

analysis of their content. The debriefings were recorded, but a thorough review was not

envisaged. These recordings were used only in the event of doubts relating to certain

situations or explanations.

The qualitative and quantitative data were collated on the basis of the phases of the

scenario and the constraints encountered in the course of the scenarios.

Data were analysed on the following basis:

By the controller and by pilots, in order to find behavioural invariants between

controllers, their analyses and their understanding of the scenario's various

phases.

By scenario phase, in order to search for usage invariants associated with

situational constraints.

The human performance results were analysed in relation to the system performance

results, as shown in Table 1, in order to analyse the CoO acceptability for the ATCOs,

considering the criterion of future traffic growth to 2020. This level of analysis is also

very relevant to understand the advantages and deficiencies of the CoO concept from an

operational point of view. The results have identified promising ways to adapt the

concept, assess its compliance with the SESAR objectives, and refine the pilots' and

ATCOs' working methods and the HMI proposed for the following experiments on the

CWP and into the cockpit.

http://en.wikipedia.org/wiki/Non-parametric_statistics

http://en.wikipedia.org/wiki/Sampling_%28statistics%29




5 HIL2 Results

The HIL2 results are presented in the light of the experiment objectives relating to

human performance and system performance, as explained in the Validation Strategy

[3].

From the stakeholders‟ concerns and performances framework [10], four of the SESAR

KPAs [6] were identified as potentially open to improvement through the introduction of

the CoO.

These KPAs are:

Capacity,

Safety,

Efficiency,

Predictability.

These four KPAs represent the system performance objectives. The aim of this validation

exercise was to assess whether the benefits were as marked as suggested. In order to

meet the objectives specific to CATS itself, it was considered worthwhile determining

whether the human contribution to the overall system performance remained within the

expected capabilities and did not reach human limits for ATCOs and pilots.

Five other areas, also called human performance objectives, were studied:

Workload,

Situation awareness,

Human performance,

Working methods,

Acceptability and usability.

Data were collected through observations, debriefings, automatic data recording, and

self-assessment questionnaires.

In addition, to improve approach and methods, the simulation realism was also assessed.

5.1 Simulation Facilities

5.1.1 Air Traffic Controller platform

On a self-assessment five-level scale, ranging from strongly disagree (level 1) to strongly

agree (level 5), the four controllers assessed:

The experimental platform and simulation facilities as high (level 4 - agree),

reaching a good level of understanding of the interactions between ANSPs, and

between the controllers and pilots, due to the Contract of Objectives and Target

Windows.

The traffic samples were found to be appropriate (level 3.25 – slightly agree) to

test the interactions between ANSPs, and between the controllers and the pilots,

due to the Contract of Objectives.




The training and familiarization received prior to the measured exercises were

sufficient (level 4.80 – strongly agree).

The new coordination tools between the planner and the executive were considered very

efficient. However, controllers proposed to improve the aircraft data display (TW and

flight plan) to cope with traffic growth to 2020.

5.1.2 Cockpit platform

On a self-assessment five-level scale, ranging from strongly disagree (level 1) to strongly

agree (level 5), the two pilots assessed:

The experimental platform and simulation facilities as high (level 4 - agree),

reaching a good level of understanding of the interactions between controllers

and pilots, due to the Contract of Objectives and Target Windows.

A two-seat flight simulator is not required to evaluate the impacts of Target

Windows on aircrew's activity (level 4 –agree).

The piloted flights were appropriate for testing interactions between controllers

and pilots (level 4 – agree).

The training and familiarization received prior to the measured exercises were

sufficient (level 4 – agree).

The way the TW data are displayed in the cockpit is satisfactory even though some

improvements have to be considered. Pilots found the lack of air traffic displaying on the

Traffic Collision Avoidance System (TCAS) contributed to reduce their situation

awareness about the traffic and the controllers' task load. This was reinforced by the lack

of party line communications between controllers and pilots, due to the fact that the

controllers and non-piloted aircraft communicated through data link (and not by Radio-

Telephony).

5.2 Human Performance Results

The human performances were assessed by different criteria (mainly workload and

situation awareness), using objective and subjective data.

Assessment was twofold: on the one hand for the controllers, and on the other hand for

the pilots.

5.2.1 Controllers' assessment of human performance

5.2.1.1 Workload

The controller‟s workload was assessed through two subjective methods: Instantaneous

Self-Assessment of Workload (ISA) and NASA-Task Load indeX (NASA-TLX) [13]. The

workload assessment purpose was to measure the impact of Target Window (TW)

management on the controller‟s workload. The way to assess this assumption was to

compare two similar traffic management situations: one without TWs and one with TWs.

At the end of each run, a post-run questionnaire also tackled this issue.




5.2.1.1.1 ISA Assessment

The ISA method was used to measure workload throughout the simulation exercise by

asking the controller to assess his (her) level of workload on a five-level scale (from level

1 – very low - to level 5 – very high). The controllers were asked to assess their level of

workload every 5 minutes. It was then possible to determine the perceived level of

workload for each controller during the current (2008) and 2020 traffic loads and at each

control position (executive or planner). The value for the workload throughout the traffic

load phase is the average of the ISA assessments during this phase.

Eight measurements were gathered for each of the six experimental conditions (2008

and 2020 traffic loads „with‟ and „without TWs‟, and 2020 traffic conditions with and

without event). A Wilcoxon Statistics non-parametric test [15] was applied to determine

whether or not there is a significant effect by means of the p-value (the probability of

obtaining a result at least as extreme as the one that was actually observed, given that

the null hypothesis is true).

Table 7 to Table 10 show the ISA results for the executive and PLNs of the KL (Geneva)

and MI (Milan) sectors in the four experimental conditions:

2008 current traffic,

2008 current traffic with TW,

2020 traffic, and

2020 traffic with TW.

0.0

1.0

2.0

3.0

4.0

5.0

2008 2008-TW 2020 2020-TW

ISAWorkLoad KL1_EXE

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.0502008/2008-TW

2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:0.00)

p = 0.148 (z:1.05)

p = 0.018 (z:-2.10)

p = 0.047 (z:-1.68)

Table 7: ISA Results for the KL (Geneva) EXE

http://en.wikipedia.org/wiki/Probability

http://en.wikipedia.org/wiki/Null_hypothesis




0.0

1.0

2.0

3.0

4.0

5.0

2008 2008-TW 2020 2020-TW

ISAWorkLoad MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:0.26)

p = 0.065 (z:-1.52)

p = 0.033 (z:-1.84)

p = 0.002 (z:-2.84)

Table 8: ISA Results for the MI (Milan) EXE

0.0

1.0

2.0

3.0

4.0

5.0

2008 2008-TW 2020 2020-TW

ISAWorkLoad KL1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.216 (z:-0.79)

p > 0.250 (z:-0.05)

p = 0.033 (z:-1.84)

p = 0.021 (z:-2.05)

Table 9: ISA Results for the KL (Geneva) PLN




0.0

1.0

2.0

3.0

4.0

5.0

2008 2008-TW 2020 2020-TW

ISAWorkLoad MI1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:0.53)

p = 0.232 (z:-0.74)

p = 0.005 (z:-2.57)

p = 0.021 (z:-2.05)

Table 10: ISA Results for the MI (Milan) PLN

Table 11 to Table 14 show the ISA results for the executive and PLNs of the KL (Geneva)

and MI (Milan) sectors with 2020 traffic conditions with or without event:

2020 traffic,

2020 traffic with TW,

2020 traffic with event,

2020 traffic with TW and event.




0.0

1.0

2.0

3.0

4.0

5.0

2020 2020-TW 2020EV 2020EV-TW

ISAWorkLoad KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.148 (z:1.05)

p > 0.250 (z:0.47)

p > 0.250 (z:0.16)

p > 0.250 (z:-0.47)

Table 11: ISA Results for the KL (Geneva) EXE with event




0.0

1.0

2.0

3.0

4.0

5.0

2020 2020-TW 2020EV 2020EV-TW

ISAWorkLoad KL1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.05)

p > 0.250 (z:-0.47)

p > 0.250 (z:-0.47)

p = 0.136 (z:-1.10)

Table 12: ISA Results for the KL (Geneva) Planning Controller with event




0.0

1.0

2.0

3.0

4.0

5.0

2020 2020-TW 2020EV 2020EV-TW

ISAWorkLoad MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.065 (z:-1.52)

p = 0.078 (z:-1.42)

p > 0.250 (z:0.26)

p > 0.250 (z:0.53)

Table 13: ISA Results for the MI (Milan) EXE with event




0.0

1.0

2.0

3.0

4.0

5.0

2020 2020-TW 2020EV 2020EV-TW

ISAWorkLoad MI1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.232 (z:-0.74)

p > 0.250 (z:0.47)

p > 0.250 (z:0.00)

p = 0.136 (z:1.10)

Table 14: ISA Results for the MI (Milan) PLN with event

The controllers' ISA ratings show the following results.

„Without event‟ conditions

There is no significant difference (p<0.05) between the „without TW‟ and „with TW‟

conditions whatever the traffic load (p values range from 0.065 to >0.25). This result is

observed whatever the control position (executive or planner) and whatever the

controlled sector (KL or MI).

A significant difference (p<0.05) is observed between the two traffic load conditions

whatever the control position and the controlled sector. The p values range from 0.002 to

0.047.

In addition, the analysis of median values show the workload of 2008 traffic load

conditions is always lower than the 2020 traffic load conditions.

These results show heavy traffic loads (2020 forecast) impact the controller workload

more than TW management.

The impact of „2020 with TW management‟ conditions is similar whatever the position

assigned to the controller (executive or planner).

„With event' conditions

There is no significant difference (p<0.05) between „with event‟ and „without event‟ in

the case of the 2020 traffic conditions, whatever the control position (executive or

planner), and whatever the controlled sector (KL or MI). This result is surprising because

the occurrence of CB is always very disturbing for controllers.




The debriefings with controllers after the exercises established that only two or three

aircraft were impacted by the CB during the scenario. In addition, it was easy to

anticipate the change of the route. The consequence was that the CB was not difficult to

manage in view of the traffic features. The changes of routes were not sufficient to

increase the resulting load for TW management.

5.2.1.1.2 NASA-TLX Assessment

NASA-TLX data were collected at the end of each simulation exercise to record the

general workload perception along the different phases of the exercises. The HIL2

protocol distinguishes 2 traffic load phases (2008 and 2020) in the exercises, and these

two phases were assessed at the end of each simulation run. In the 2020 traffic load

phases, the „without event‟ and „with event‟ conditions were also systematically assessed

at the end of the exercise.

The NASA-TLX assessment values range from 0 (the lowest level of workload) to 100

(the highest level of workload).



without event), and a Wilcoxon statistical test was applied to determine whether or not

there is a significant effect.

Table 15 to Table 18 below show the NASA-TLX results for the executive and PLNs on the

KL (Geneva) and MI (Milan) sectors in the four experimental conditions:



2020 traffic, and





0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2008 2008-TW 2020 2020-TW

NASA TLX WorkLoad KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.174 (z:-0.95)

p > 0.250 (z:-0.32)

p < 0.002 (z:-3.36)

p < 0.002 (z:-2.94)

Table 15: NASA-TLX Results for the KL (Geneva) EXE

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2008 2008-TW 2020 2020-TW

NASA TLX WorkLoad KL1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:-0.63)

p = 0.174 (z:0.95)

p < 0.002 (z:-2.94)

p = 0.023 (z:-2.00)

Table 16: NASA-TLX Results for the KL (Geneva) PLN




0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2008 2008-TW 2020 2020-TW

NASA TLX WorkLoad MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:0.32)

p > 0.250 (z:0.32)

p < 0.002 (z:-2.94)

p < 0.002 (z:-3.36)

Table 17: NASA-TLX Results for the MI (Milan) EXE

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2008 2008-TW 2020 2020-TW

NASA TLX WorkLoad MI1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:-0.63)

p = 0.114 (z:-1.21)

p = 0.006 (z:-2.52)

p < 0.002 (z:-2.99)

Table 18: NASA-TLX Results for the MI (Milan) PLN




Table 19 to Table 22 show the NASA-TLX results for the executive and PLNs of the KL

(Geneva) and MI (Milan) sectors in 2020 traffic conditions, with or without event:

2020 traffic,




0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2020 2020-TW 2020EV 2020EV-TW

NASA TLX WorkLoad KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.32)

p > 0.250 (z:-0.32)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 19: NASA-TLX Results for the KL (Geneva) EXE with event




0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2020 2020-TW 2020EV 2020EV-TW

NASA TLX WorkLoad KL1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.174 (z:0.95)

p = 0.174 (z:0.95)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 20: NASA-TLX Results for the KL (Geneva) PLN with event




0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2020 2020-TW 2020EV 2020EV-TW

NASA TLX WorkLoad MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:0.32)

p > 0.250 (z:0.32)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 21: NASA-TLX Results for the MI (Milan) EXE with event




0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

2020 2020-TW 2020EV 2020EV-TW

NASA TLX WorkLoad MI1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.114 (z:-1.21)

p = 0.114 (z:-1.21)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 22: NASA-TLX Results for the MI (Milan) PLN with event

The controllers' NASA-TLX ratings show the following results.



conditions whatever the traffic load (p values range from 0.114 to >0.25). This result is

observed whatever the control position (executive or planner) and whatever the

controlled sector (KL or MI).

The significant difference (p<0.05) is between the two traffic load conditions whatever

the control position and the controlled sector. The p values range from <0.002 to 0.006.

In addition, the analysis of median values shows the workload of 2008 traffic load

conditions was always lower than the 2020 traffic load conditions.

These results show heavy traffic loads (2020 forecast) impacted the controller workload

more than TW management.

The impact of „2020 with TW management‟ condition is similar whatever the position

assigned to the controller (executive or planner).

"With event' conditions

There is no significant difference (p<0.05) between „with event‟ and „without event‟ in

the case of the 2020 traffic conditions, whatever the control position (executive or

planner), and whatever the controlled sector (KL or MI). This result is similar with the

ISA ratings and the explanation is the same as with ISA.




5.2.1.1.3 Post-run Questionnaire and Debriefing Data

The four controllers felt that Target Windows management increases the workload in

2020 conditions. They didn't feel the same for the 2008 conditions. This perception is

slightly higher for the executive than for the planner position. The perception of higher

workload is strongly linked to the way the Target Window data are displayed on the

screen, and the way of perceiving them.

The impact of „with event‟ condition exercises has already been discussed in the ISA

section.

5.2.1.1.4 Workload Conclusion

The HIL2 data showed that the workload perception was slightly impacted with high

traffic load (2020 forecast), and this also impacted their perception of workload in „TW‟

conditions.

The ISA and NASA-TLX measurements are consistent and provide similar results,

strengthening their validity.

These subjective data (ISA and NASA-TLX) did not show any significant statistical

difference between the „2020 without TW‟ conditions and „2020 with TW‟ conditions and

with event or without event. However, the „with event‟ condition is probably not fully

representative of the true impact of a CB event.

The workload assessment showed that the workload was not statistically impacted by the

management of the TWs whatever the traffic load, although the controllers perceived the

TWs as an additional task, slightly impacting traffic management. An explanation of this

feeling may be the use of the current route structure, not fully adapted to the use of TWs

with heavy traffic. This may also be due to the need for controllers to change the way

they manage the traffic by shifting from a local view (sector management) to a system

integrated view, in which the TW concept is developed.

These results are consistent with the HIL1 results. They confirm the tendency that Target

Windows are felt to be an additional load whose impact is perceived as a strain when

traffic loads are high. However, the measured workload does not show a significant

difference in high traffic conditions between Target Windows and without Target

Windows. This means that the experimental conditions probably reach the limits and the

HMI design is very important in promoting controller acceptance.

5.2.1.2 Situation Awareness

Situation awareness is a factor enabling the controllers to understand the traffic

dynamics and to be able to properly manage the aircraft in order to ensure safety and

efficiency.

The situation awareness, also called the "picture", is defined by Endsley [12] as the

perception of the elements in the environment within a volume of time and space, the

comprehension of their meaning and the projection of their status in the near future.

The situation awareness was evaluated in the HIL 2 experiment through SASHA

questionnaires [14] and also tackled during post–run questionnaires.




5.2.1.2.1 SASHA-Q Assessment

Within the SASHA questionnaire (SASHA-Q), six questions allow the controllers to assess

situation awareness by using a seven-point scale ranging from 0 (very low) to 6 (very

high).

The definitive situation awareness score is then obtained by calculating the average of all

question scores.



without event), and a Wilcoxon statistical test was applied to determine whether or not

there is a significant effect.

Table 23 to Table 26 show the SASHA-Q results for the executive and PLNs on the KL

(Geneva) and MI (Milan) sectors in the four experimental conditions:



2020 traffic, and

2020 traffic with TW

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2008 2008-TW 2020 2020-TW

SASHA (Situational Awareness for SHAPE) KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:0.00)

p > 0.250 (z:-0.11)

p < 0.002 (z:3.15)

p < 0.002 (z:3.05)

Table 23: SASHA-Q Results for the KL (Geneva) EXE




0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2008 2008-TW 2020 2020-TW

SASHA (Situational Awareness for SHAPE) KL1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.124 (z:-1.16)

p = 0.187 (z:-0.89)

p < 0.002 (z:3.26)

p = 0.002 (z:2.84)

Table 24: SASHA-Q Results for the KL (Geneva) PLN

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2008 2008-TW 2020 2020-TW

SASHA (Situational Awareness for SHAPE) MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p > 0.250 (z:-0.05)

p > 0.250 (z:-0.05)

p = 0.004 (z:2.68)

p = 0.007 (z:2.47)

Table 25: SASHA-Q Results for the MI (Milan) EXE




0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2008 2008-TW 2020 2020-TW

SASHA (Situational Awareness for SHAPE) MI1_PLN

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- Impacted --

-- Impacted --

-- Impacted --

-- No Impact --

p = 0.026 (z:1.94)

p = 0.006 (z:2.52)

p = 0.018 (z:2.10)

p = 0.065 (z:1.52)

Table 26: SASHA-Q Results for the MI (Milan) PLN

Table 27 to Table 30 show the SASHA-Q results for the executive and PLNs of the KL

(Geneva) and MI (Milan) sectors in 2020 traffic conditions with or without event:

2020 traffic,







0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW 2020EV 2020EV-TW

SASHA (Situational Awareness for SHAPE) KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.11)

p > 0.250 (z:-0.11)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 27: SASHA-Q Results for the KL (Geneva) EXE with event

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW 2020EV 2020EV-TW

SASHA (Situational Awareness for SHAPE) KL1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.187 (z:-0.89)

p = 0.187 (z:-0.89)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 28: SASHA-Q Results for the KL (Geneva) PLN with event




0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW 2020EV 2020EV-TW

SASHA (Situational Awareness for SHAPE) MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.05)

p > 0.250 (z:-0.05)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 29: SASHA-Q Results for the MI (Milan) EXE with event

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW 2020EV 2020EV-TW

SASHA (Situational Awareness for SHAPE) MI1_PLN

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- Impacted --

-- Impacted --

-- No Impact --

-- No Impact --

p = 0.006 (z:2.52)

p = 0.006 (z:2.52)

p > 0.250 (z:0.00)

p > 0.250 (z:0.00)

Table 30: SASHA-Q Results for the MI (Milan) PLN with event




The controllers' SASHA-Q ratings show the following results.


For the KL (Geneva) sector planner and EXEs and the MI (Milan) EXE, there is no

significant difference (p<0.05) between the „without TW‟ and „with TW‟ conditions

whatever the traffic loads (p values range from 0.124 to >0.25). For these 3 positions,

the significant difference (p<0.05) is between the two traffic load conditions whatever

the control position and the controlled sector. The p values range from <0.002 to 0.007.

In addition, for these three positions the analysis of median values shows the situation

awareness of 2008 traffic load conditions has been always higher than the 2020 traffic

load conditions.

These results show heavy traffic loads (2020 forecast) impacted the controller situation

awareness more than TW management.

For the MI (Milan) PLN, there is a significant difference (p<0.05) between the „without

TW‟ and „with TW‟ conditions, whatever the traffic loads. There is also a significant

difference between the two traffic conditions (2008 and 2020) without TW. For the two

same traffic conditions, but with TW, there is no significant difference (p>0.05). Such

results are not easy to interpret, and Target Windows use does not seem to be the main

factor for these differences. It is more probable that the origin of these differences is to

be found in the airspace features.

„With event' conditions

For the KL (Geneva) sector planner and EXEs and the MI (Milan) EXE, there is no

significant difference (p<0.05) between the „with event and the „without event‟ conditions

in the 2020 traffic conditions.

For the MI (Milan) PLN, there is significant difference (p<0.05) between the „without TW‟

and „with TW‟ conditions whatever the „with event‟ or „without event‟ conditions. Here,

TW use has a significant effect on the situation awareness - it slightly decreases it. The

decreased level of situation awareness is a high level of situation awareness (the lower

median values being 5.3 and 5.4 on a scale between 0 and 6).


The controllers' feeling about the impact of TW management on the situation awareness

is unanimous: the use of TWs does not decrease traffic situation awareness. All the four

controllers agree and chose to rate (level 4.2 on the five-level scale) situation awareness

as increasing with TWs. This feeling is of the same order for the EXE and the planner

position.

The explanation given by the controllers during debriefings is that the information

displayed about TWs is very useful for aircraft management. However, controllers find it

challenging to manage TW information when traffic load becomes heavy. They stress the

need to better integrate TW information in the aircraft flight plan data displayed on the

radar screen. In particular, they need to have a long=term understanding of TW

requirements.

5.2.1.2.3 Situation Awareness Conclusion

Regarding the quantitative data, situation awareness is not impaired by the

implementation of TWs, although the results for one position (MI – PLN) do not comply.




Traffic load impacted situation awareness but TW management did not for the KL

(Geneva) sector planner and EXEs and the MI (Milan) EXE. The quantitative data for the

MI (Milan) PLN show that TWs slightly impacted the situation awareness, although other

factors such as the sector airspace could influence how the planner built his/her picture.

The results for "with event" conditions reinforce this feeling.

The controllers' feeling on situation awareness is that TW information increases the traffic

picture, although there was no statistical significant difference evident with situation

awareness quantitative data for the KL (Geneva) sector planner and EXEs and the MI

(Milan) EXE, and if the MI (Milan) EXE's situation awareness is significantly impaired.

This feeling was justified by the specific information displayed for the TWs. This

information allowed the controller to manage the flight plan better than with the

information currently displayed. TW information gave the controller more details,

particularly regarding the exit conditions, to deal efficiently with the traffic taking into

account the full constraints of the flight (not only the local sector constraints), although

controllers said improvements had to be made with a view to ensuring better integration

of TW data with flight plan data.

5.2.1.3 Human Performance

Human performance was assessed by the over-the-shoulder method, post-run

questionnaire and debriefing.

5.2.1.3.1 Over-the-shoulder method

The performances were assessed by Human Factors experts. They used the over-the-

shoulder (OTS) method developed by FAA. The controllers‟ performances were assessed

for the pair of controllers (planner and executive) on each sector. The focus was put on

the „2020 traffic load‟ conditions.

The FAA OTS method was used to assess the quality of different aspects of controller

performance. The original method was adapted to the specifics of TW management and

the limitations of the platform simulation. Then, certain items were cancelled and others

were added. The HIL2 OTS grid comprises the following factors:

Maintaining separation

Maintaining efficient traffic flow

Maintaining attention and situation awareness

Coordinating

Communicating and cooperation

Performing multiple tasks

Managing sector workload

Overall Performance

Each factor was illustrated by a set of sub-topics to increase the reliability and validity of

the observations. The factors were rated on a seven-point scale ranging from 1 (low) to 7

(exceptional).

Table 31 and Table 32 illustrate the OTS „overall performance‟ on the two measured

sectors for „2020 traffic load‟ conditions. The other factors are consistent with the „overall

performance‟ factor and are not displayed here.




Eight measurements were made for the two conditions on each measured sector (KL:

Geneva and MI: Milan). The Wilcoxon statistical test was applied to determine whether or

not there is a significant effect when using TWs.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW

OTS overall Performance 2020 (Over The sholders observations) KL1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.050

2020/2020-TW -- No Impact -- p > 0.250 (z:-0.37)

Table 31: OTS "Overall Performance" results for the KL (Geneva) Sector




0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

2020 2020-TW

OTS overall Performance 2020 (Over The sholders observations) MI1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.050

2020/2020-TW -- No Impact -- p > 0.250 (z:0.00)

Table 32: OTS "Overall Performance" results for the MI (Milan) Sector

There is no significant difference (p<0.05) of „Overall Performance‟ level between

„without TWs‟ and „with TWs‟ in the case of „2020 traffic load‟ conditions. This result is

observed whatever the controlled sector - KL or MI (p values are >250). The TWs do not

impact the overall performance of the controller pair whatever the controlled sector in

the case of the 2020 traffic load.

The analysis of other „performance‟ elements assessed through the OTS grid showed

similar results. There is no statistically significant difference (p<0.05) between „without

TW‟ and „with TW‟ in the case of the 2020 traffic load.


Controllers confirmed that their performances were not impaired by TW management.

They didn't experience any difficulties managing the TWs, except when the TWs are too

close to the sector boundaries. This is true for adjacent and superimposed Target

Windows. This difficulty was due to the calculation of TWs and was a simulation

limitation. A solution for this impairment will be found through the calculation of Target

Windows.

Controllers agreed that TWs cannot increase controller performance. TW benefits are

more for the airlines than for the controllers.

5.2.1.3.3 Human Performance Conclusion

The qualitative data did not show a significant difference in Human Performance between

„TW‟ conditions and „non TW‟ conditions. This result confirmed the controllers' feeling that

the same level of performance was achieved when using TWs.




However, the acceptance by controllers of TWs with high traffic load required an

adjustment to the controller working position HMI in the form of additional support tools.

5.2.1.4 Working Methods

5.2.1.4.1 TW management

TW management was not a difficult task for the controllers. It was easy for them to

systematically pick up the TW information when the aircraft were accepted. Only when

the traffic load was heavy did the controllers encounter some difficulties picking up this

information. This drawback was not specific to TW data - it was also observed for the

other flight plan information. It was mainly related to the HMI layout and heavy traffic

load. However, all controllers stress the need to improve the HMI in such control

situations.

The only change to their current way of working was that they gave different direct

orders in order to respect the exit TWs. Their direct orders diverged less from the initial

route than usual. This holds true for all controllers, but the way they did this depends on

the individual controller‟s or ACC‟s way of working.

5.2.1.4.2 Traffic Understanding and Anticipation

Controllers agreed that TWs were systematically used to understand and anticipate the

traffic. Regarding the 2 traffic loads used for the simulation exercises, they had difficulty

remaining aware of TW information with 2008 current traffic load. The difficulty was more

with the high 2020 traffic load, because the amount of information to process in a short

period of time was high.

TW management was judged to be a task mainly for the PLN. Information brought by the

TWs is firstly long-term, contributing to traffic efficiency. PLN was in charge of pre-sector

planning and preparation. This task was enhanced by the additional information provided

by the TWs. PLN was responsible for preparing the instructions to be given by the

executive with a view to fulfilling the exit TWs. The role of the executive, regarding the

TW management task, was mainly prepared and anticipated by the planner. This is

largely why collaboration between executive and PLN is very important with TWs.

The specific collaboration tools designed for the HIL2 experiment are judged satisfactory

by the controllers. This tool allows the planner to propose aircraft route solutions to the

executive, mainly under the 2020 traffic conditions, where the time for discussion was

drastically reduced. The collaborative tool considerably improves collaboration and

contributes to better coordination between the executive and the planner. They also help

to better define the role of each controller in the management of TWs.

TWs may help with short-term management, but safety remained highest priority and

sometimes safety constraints were given precedence over TW constraints. This aspect

was given particular emphasis by the controllers during the debriefings.

5.2.1.4.3 Conflict Detection and Solving

The way the executive and the PLNs detect conflict is judged to be similar with TW

management (level 4 on the five-point scale).

The way the EXE solved the conflicts is judged to be different with TW implementation

(level 4 on the five-point scale). Such a change is linked to the objective of fulfilling the

exit TW and finding the best solution to solve the conflict in compliance with this

constraint, if possible. When safety may be impaired, the executive systematically




applies a separation solution without considering the TW constraints. This reinforces the

fact that at no time do TWs take precedence over safety concerns in the controller‟s

mind.

After applying the separation solution, the EXE tries where possible to fulfil TWs.

Nevertheless, with more practice, EXEs confirmed that they could develop skills for better

integration of TW constraints in the conflict solving procedures. No changes of practice

(e.g. vertical instruction rather than lateral) were observed during the experiment.

For PLNs, no differences were identified between „without‟ or „with TW‟ . TWs were an

added source of information, integrated easily by the planner in his (her) current working

methods for detecting conflicts.

These results were confirmed by the self-assessment regarding the „overall subjective

risk level‟ and if „they experienced hazardous situations along the simulation exercise‟.

Whatever the „TW‟ condition, controllers kept a high level of safety comfort and

encountered very few situations which may result in a hazardous situation.

The „what if‟ tool, designed to help the controllers in their decision to manage safety

while fulfilling the TWs was used in different ways by the controllers. Some used it and

others not, although they said in the debriefing that the tool was useful. The controllers

who used it said it was useful and needed to be coupled with a medium-term conflict

detection system. In this case, the tool is essential for managing heavy traffic load. The

controllers who do not use the „what if‟ tool explained that:

In low traffic load, they had the TW situation awareness, so there was no need

for them to cross-check with the tool.

In heavy traffic load, they forget to use it on account of lack of practice.

5.2.1.4.4 Controllers' Orders

Controllers' orders were analyzed in order to understand the potential impact of TWs on

the way the EXEs managed the traffic. Only the orders linked to aircraft trajectory were

studied. The handover orders between ANSPs were not recorded and studied.

The controllers' orders encompass different orders to aircraft, namely:

Heading

Flight level

Speed

Direct to ("go to")

Firstly, the orders were studied as a whole. Tables 33 and 34 show all the orders given

by the EXE in each sector for the four experimental conditions. The orders recorded

during the different phases of runs were processed to be presented as the number of

orders given per hour for the same reference number of aircraft in each condition. Only

then is it possible to compare the two traffic load conditions.



without event), and a Wilcoxon statistical test was applied to determine whether there is

a significant effect, or not.

Table 33 and Table 34 show the global orders results for the EXEs on the KL (Geneva)

and MI (Milan) sectors in the four experimental conditions:






2020 traffic, and

2020 traffic with TW

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2008 2008-TW 2020 2020-TW

ATCO GLOBAL Orders/hours KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.161 (z:1.00)

p = 0.201 (z:0.84)

p = 0.008 (z:-2.42)

p = 0.037 (z:-1.79)

Table 33: Total Number of Controllers' Orders for the KL (Geneva) Sector




0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2008 2008-TW 2020 2020-TW

ATCO GLOBAL Orders/hours MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.124 (z:1.16)

p = 0.174 (z:0.95)

p = 0.148 (z:-1.05)

p = 0.201 (z:-0.84)

Table 34: Total Number of Controllers' Orders for the MI (Milan) Sector

Table 35 and Table 36 show the global orders results for the EXEs of the KL (Geneva)

and MI (Milan) sectors in 2020 traffic conditions with or without event:

2020 traffic,







0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2020 2020-TW 2020EV 2020EV-TW

ATCO GLOBAL Orders/hours KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.201 (z:0.84)

p > 0.250 (z:0.53)

p = 0.174 (z:0.95)

p = 0.216 (z:0.79)

Table 35: Total Number of Controllers' Orders for the KL (Geneva) Sector with event

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2020 2020-TW 2020EV 2020EV-TW

ATCO GLOBAL Orders/hours MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- Impacted --

p = 0.174 (z:0.95)

p > 0.250 (z:-0.26)

p = 0.201 (z:-0.84)

p = 0.033 (z:-1.84)

Table 36: Total Number of Controllers' Orders for the MI (Milan) Sector with event




The EXEs' global orders data show the following results.



conditions whatever the traffic load (p values range from 0.84 to 0.174).

A significant difference (p<0.05) was found between the two traffic load conditions for

the KL (Geneva) sector.

These results show that the traffic load impacts the way the KL (Geneva) EXE manage

the traffic. With more traffic, more orders are given.

This is not true for the MI (Milan) sector, where there is no significant difference between

the traffic loads. The difference between the two sectors is probably due to the sector

size and airspace features. The KL sector is smaller than the MI sector. Consequently,

EXEs work more in a reactive way and give more orders to aircraft.

Target Windows management does not impact the number of orders given to aircraft,

whatever the controlled sector.

„With event‟ conditions

There is no significant difference (p<0.05) between „without TW‟ and „with TW‟ conditions

irrespective of the „with event‟ conditions (p values range from 0.174 to >0.250),


A significant difference (p<0.05) is observed for the MI (Milan) sector with Target

Windows conditions between „with event‟ and „without event‟ conditions. The location of

the event within the sector in relation to the airways is probably the cause of this result.

This result cannot be explained by the presence of TW conditions.

The details are subsequently analysed if there is difference regarding each type of

instruction. The idea is to see whether TWs changed the type of orders given by the

ATCOs to the aircraft.

Table 37 to Table 48 show the number of each type of orders, in each sector and for each

traffic load conditions. Each type of instruction, recorded during the different phases of

the simulation runs, was processed to be presented in terms of the number of orders

given per hour, to allow comparison.

The results are displayed in the following sequence:

FL orders;

Speed orders;

"Go to" orders.

Firstly the results are displayed for the „without‟ event conditions, and secondly for the

„with event‟ conditions.

Regarding the simulation limitations, the controller cannot give heading orders to the

aircraft. The solution is to give closed orders like ""go to"" orders. This explains why

heading orders are not counted here.





0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO FL Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.174 (z:-0.95)

p > 0.250 (z:-0.32)

p = 0.002 (z:-2.84)

p = 0.006 (z:-2.52)

Table 37: Flight Level Orders for the KL Sector

0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO FL Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- Impacted --

-- No Impact --

p > 0.250 (z:-0.32)

p = 0.187 (z:0.89)

p = 0.047 (z:-1.68)

p > 0.250 (z:0.11)




Table 38: Flight Level Orders for the MI Sector

0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO SPEED Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.124 (z:1.16)

p > 0.250 (z:-0.37)

p = 0.148 (z:1.05)

p > 0.250 (z:-0.53)

Table 39: Speed Orders for the KL Sector

0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO SPEED Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.161 (z:-1.00)

p > 0.250 (z:-0.11)

p = 0.161 (z:-1.00)

p > 0.250 (z:-0.16)




Table 40: Speed Orders for the MI Sector

0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO GOTO Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- Impacted --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.047 (z:1.68)

p = 0.058 (z:1.58)

p = 0.105 (z:-1.26)

p = 0.174 (z:-0.95)

Table 41: "Go to" Orders for the KL Sector




0.0

20.0

40.0

60.0

80.0

100.0

2008 2008-TW 2020 2020-TW

ATCO GOTO Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- Impacted --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.026 (z:1.94)

p = 0.232 (z:0.74)

p = 0.232 (z:0.74)

p = 0.174 (z:-0.95)

Table 42: "Go to" Orders for the MI Sector





0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO FL Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.32)

p > 0.250 (z:-0.47)

p = 0.065 (z:1.52)

p = 0.174 (z:0.95)

Table 43: Flight Level Orders for the KL Sector with event




0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO FL Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.187 (z:0.89)

p = 0.216 (z:-0.79)

p > 0.250 (z:0.47)

p = 0.078 (z:-1.42)

Table 44: Flight Level Orders for the MI Sector with event

0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO GOTO Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- Impacted --

-- No Impact --

-- No Impact --

p = 0.058 (z:1.58)

p = 0.030 (z:1.89)

p > 0.250 (z:-0.16)

p > 0.250 (z:0.05)

Table 45: "Go to" Orders for the KL Sector with event




0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO GOTO Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.232 (z:0.74)

p = 0.096 (z:1.31)

p = 0.124 (z:-1.16)

p = 0.148 (z:-1.05)

Table 46: "Go to" Orders for the MI Sector with event

0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO SPEED Orders/Hours KL1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.37)

p = 0.174 (z:-0.95)

p > 0.250 (z:0.47)

p > 0.250 (z:-0.21)

Table 47: Speed Orders for the KL Sector with event




0.0

20.0

40.0

60.0

80.0

100.0

2020 2020-TW 2020EV 2020EV-TW

ATCO SPEED Orders/Hours MI1_EXE

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.11)

p = 0.248 (z:-0.68)

p = 0.248 (z:0.68)

p > 0.250 (z:0.16)

Table 48: Speed Orders for the MI Sector with event

Eight measurements were made for each experimental condition, on each measured

sector (KL: Geneva and MI: Milan). The Wilcoxon statistical test was applied to determine

whether or not TW use has a significant effect.

Flight level orders. TW management does not impact the number of flight level

orders whatever the traffic load. The number of flight level orders is impacted by

the traffic load for the KL (Geneva) sector with or without TWs. For the MI

(Milan) sector, the flight level orders are impacted only by the traffic load

without TWs. In the two sectors, the median is higher for the 2020 traffic

conditions than for the 2008 traffic conditions. These results mean the number of

flight level orders is greater when the traffic increases. In the „with event‟

conditions, there is no impact for the two sectors on the number of flight level

orders whatever the TW and traffic load conditions.

Speed orders. There is no impact of TWs and traffic load on the number of speed

orders. Speed orders do not create sufficient room for manoeuvre, because of

the length of the sectors. This result is similar for the „with event conditions‟.

"Go to" orders. The "go to" orders are significantly less likely with TWs in 2008

traffic load condition for both sectors. For the 2020 traffic conditions, the orders

are also less likely but this difference is not significant. These results confirm

that the controllers are managing the traffic by giving, when they can, "go to"

orders to the aircraft. The TWs impose constraints which limit the possibility to




give "go to" orders. When the traffic load increases, "go to" orders are more

difficult to give because of the higher risk of loss of separation. In „with event‟

conditions, there is no significant difference between TWs or no TWs, for the MI

(Milan) sector EXE. For the KL (Geneva) sector EXE, the significant difference is

between whether there is TW management or not (less "go to" orders are given

with TWs). Such a difference probably results from the constraints (small sector

size, heavy traffic, and event) which limit the room for manoeuvre in the

horizontal dimension. Consequently, controllers instruct the aircraft to move

vertically. Of course, the accumulation of constraints reduces options for

managing the traffic and TW use impacts orders given by the controller.

The analysis of the number of orders given shows that the working methods are slightly

impacted by TW use. As supposed, the number of orders increased when traffic load

increased, but independently of TW use. However, some differences are observed when

the traffic constraints increase: more flight level and less "go to" orders are given by the

controllers. The sector size and airspace impact the controllers' room for manoeuvre.

Speed orders are not used by the controllers, as they are mainly used to maintain an

achieved separation. These orders have an operational use when they can be used over

several sectors.

During the debriefing, the controllers did not feel that their use of orders was affected by

introduction of the TWs.

The differences observed in the HIL2 experiment are consistent with the way the

controllers are working when the constraints increase.

5.2.1.4.5 Collaboration between controllers

Controllers feel collaboration processes between executive and planner, and between the

two ANSPs controllers are not impacted by the use of TWs.

The HIL1 experiment lessons learned showed that collaboration between executive and

planner should be enhanced by a TW collaboration tool. Such a tool was designed and

implemented in the HIL 2 controller simulation platform. The tool takes into account:

TW management issues,

Current cooperation issues (non-specific to TWs).

The collaboration tool is very useful and widely appreciated by the controllers. Their

feeling is that the higher the constraints are, the more tools and support systems are

required. The collaboration tool proposed in HIL2 meets this requirement.

With the collaboration tool, the controllers assess that TW management does not require

more communication (level 3.5 on the five-point scale). They feel the cooperation work

between the executive and the planner is not impacted by the TWs (level 4 on the five-

point scale). They also assess the workload needed to cooperate as being slightly higher

(level 3.5 on the five-point scale), but the common picture of traffic between the

executive and the planner is not decreased (level 4 on the five-point scale).

All agree that TW use is not a concept which increases cooperation with adjoining sectors

(level 4 on the five-point scale). This does not mean that TW use impairs this

cooperation, but only that the TW concept does not improve it.

The TWs are assessed as an operational interface which is more complete than the one

provided by the current flight plan data. In this way, TWs increased operational flight

continuity between sectors and between ANSPs, thus facilitating the whole air traffic




management. These expected objectives are air traffic system objectives. Although the

controllers agree with these objectives, they do not think the concept will bring more

functional continuity and cooperation between the controllers of adjoining sectors. This

feeling is probably strongly linked to the current situation in the control room in which

controllers manage the traffic without having an overview of the traffic constraints.

5.2.1.4.6 Collective aspects between controllers and pilots

The collective aspects of the relationships between controllers and pilots are the key

issue of the HIL 2 experiment. For this reason, flights with pilots are introduced into the

traffic in order to assess the impact of collective work between controllers and pilots.

The objectives are to:

analyze communications between the ground and the aircraft ;

assess TW use and the understanding between controllers and pilots.

Table 49 shows the results for the duration of controller communication with piloted

aircraft. The duration of controller communication with aircraft is calculated as a

percentage of piloted aircraft flight time in the sectors. This means that 1.2 corresponds

to 1.2 % of the flight time of piloted aircraft in the two sectors (Geneva and Milan).

Radio Communication

Median 25%-75% Min-Max

2008 2020 2008TW 2020TW0,8

1,0

1,2

1,4

1,6

1,8

2,0

2,2




Wilcoxon test: Variable impact results if p<0.05

2008/2008-TW No impact P=0.0678

2020-2020-TW No impact P=0.2733

2008-2020 No impact P=0.1441

2008TW/2020TW No impact P=0.1441

Table 49: Duration of controller communication with piloted aircraft

There is no significant difference (p<0.05) for the duration of controller communication in

the two sectors whatever the traffic load conditions and „with‟ or „without‟ Target Window

conditions (p range from 0.06 to 0.27). Target Windows management does not impact

the way the controllers communicate with the aircrews, nor does the traffic load impact

the way the controllers communicate with the aircrews.

Whether using TWs or not, 2020 traffic conditions generate longer duration of

communication, although not significantly.

The TWs do not modify the way the controller and the pilots exchange messages and

cooperate on managing the aircraft route.

Controllers and pilots feel that TW management does not require more communications

between them. On a five-point scale, controllers and pilots assess level 4. This means

that during the exchanges between controllers and pilots, messages specifically

mentioning the TWs are scarce. This result comes from the fact that the same TW data

were shared by the cockpit and the controller working position. Consequently, all

requests or orders are understood in the context of the TW data, without specifically

referring to it. When a request or an instruction is not well understood by the controller

and/or the pilot, he/she clarifies the reference to the TWs. When the pilots and

controllers evaluate this improvement in the representation of the pilot's and controller's

intent about the aircraft trajectory, the pilots rate 3.5, and the controllers 4.3 on a five-

point scale.

Controllers and pilots have no difficulty in finding vocabulary in order to communicate

about TWs. Controllers assess this easiness of communication at level 4.5, and pilots at

level 4, on a five-point scale. Despite this positive result on the intuitiveness of the TW

concept, pilots and controllers have expressed the need for a new specific phraseology.

In order to do this, the use of terms "due to Target Windows" seems accurate. By saying

"due to Target Windows", the controller and the pilots immediately understand the

context of a request or an instruction.

By adding information on the flight plan data, TWs improve the representation of aircraft

intent that controllers and pilots share. Cooperation is thus improved between the

controllers and pilots. This impression is rated level 4 on a five-point scale by the

controllers and the pilots.

The TWs are judged positive for cooperation between controllers and pilots. The cost of

this cooperation in terms of workload is rated as being without additional workload by

controllers (level 3 on a five-point scale), and as increasing slightly the workload by pilots

(level 4 on a five-point scale).

Finally, pilots and controllers agree that in normal operative conditions, communications

about Target Windows may be supported by data-link. Voice communication has to be

kept for emergencies and situations involving a lack of understanding.




5.2.1.5 Acceptability and Usability

Unanimously, controllers strongly agreed that TWs are easy to use whatever the control

position (EXE or PLN). They rate this easiness at level 4.25 on a five-point scale. This

feeling was largely expressed during the post-run debriefings. Controllers quickly became

familiarized with the concept, and were autonomous on their controller working positions.

This feeling was reinforced by the fact the controllers found TW management easy to

learn (level 4.5 rating on a five-point scale). They were satisfied with the training and

familiarization programme. The training was evaluated as satisfactory, allowing a good

understanding of the concept and its use.

The high level of usability assessed during the simulation is a positive point of the

concept and for its future development. This means that the learning process is not a

barrier for TW experiments, and for the potential implementation of TWs in actual

operational rooms.

With high traffic loads, integration of TW data into the control working position HMI is a

key challenge for the success of TW acceptability. The right information has to be easy to

pick up without additional workload. This issue is not specific to TW management but

arises systematically when the workload is high, and with high traffic volumes.

To sum up, the TW concept is well accepted. Efforts have to be made on the TW data

display and the TW size and location on the sector. Controllers stress some aspects of the

TW position:

Two following Superimposed Target Windows should be large/long enough to

allow room for manoeuvre for operators and be optimal for aircraft navigation.

Superimposed Target Windows should not be on the main crossing points.

Superimposed Target Windows have to be far from the sector vertical

boundaries, to avoid potentially four sectors responsible for the same aircraft. A

buffer zone is really needed.

As explained in HIL1, TWs should not be at the same exit point, same FL

envelope and same time envelope.

5.2.2 Pilots' human performance assessment

5.2.2.1 Workload

The pilots' workload was assessed through the NASA-Task Load index (NASA-TLX) [13].

The purpose of workload assessment was to measure the impact of Target Window (TW)

management on the pilots‟ workload. The way to assess this assumption was to compare

two piloted flight situations: one without TWs and one with TWs. At the end of each run,

a post-run questionnaire also tackled this issue.

5.2.2.1.1 NASA-TLX Assessment

NASA-TLX data were collected at the end of each simulation exercise to record the

general workload impression for each piloted flight. The HIL2 protocol distinguishes two

conditions in the exercises for pilots:

Exercise without Target Window;

Exercise with Target Windows.




In each exercise, two piloted aircraft were systematically flown on each flight simulation

platform. Because the HIL 2 experiment has two flight simulation platforms with one

different pilot on each platform, 4 piloted aircraft are flown in each exercise.

Sixteen exercises were performed for both Target Windows conditions („with‟ and

„without‟). Then, 64 measurements were gathered for each of the two experimental

conditions, and a Wilcoxon statistical test was applied to determine whether or not there

was a significant effect.

Table 50 below shows the NASA-TLX results for the pilots „with‟ and „without TWs‟.

NASA TLX Pilots


Without TW With TW0

5

10

15

20

25

30

35

40


With-TW / Without-TW Impact P=0.00008

Table 50: NASA-TLX Results for the pilots

There is significant difference (p<0.05) between the „with TW‟ and „without TW‟

conditions. The workload is higher when the pilots have to manage Target Windows than




when they do not. Although there is a difference, the workload values are low, whatever

the TW conditions. The median values are 15 and 20, and the maximum values are 35

and 37 (the maximum workload is 100 and a high level of workload starts at 60). The

impact of TWs on the pilot workload is real, but low. However, this result needs to be

taken into account for the flight phases where the aircrew workload is already high.


Pilots rate that TW use requires mental resources and time (they rate level 4 on a five-

point scale). They agree the workload is increased when they have to manage the Target

Windows during the flight (level 4 on a five-point scale). Pilots do not find that this

workload increase is critical, except in case of an emergency situation or abnormal

procedures.

5.2.2.1.3 Workload Conclusion

The HIL 2 results show that the pilots' workload is impacted by the TWs. The impact is

low, although it is significant. This impact is fully acceptable with the workload generated

by the other cockpit tasks during the cruise phase. However, experiments will be done to

validate the workload generated by TWs in cruise emergency situations or in other flight

phases where the aircrew workload is high (e.g. descent or approach in complex

environments). Such experiments will require more sophisticated flight simulations.

5.2.2.2 Situation Awareness

Situation awareness is a factor enabling the pilot to understand aircraft status and its

future in order to ensure safety and efficiency.

The situation awareness was evaluated in the HIL 2 experiment with SASHA

questionnaires – SASHA-Q [14] and also tackled during post–run questionnaires.

5.2.2.2.1 SASHA-Q Assessment

The SASHA-Q is a tool designed for controllers. For the HIL 2 experiment, the SASHA-Q

was adapted to the pilots. The purposes of the questions are the same (same situation

awareness dimensions), but adapted to the pilot context.

Like for the workload, 16 exercises were performed for each Target Window condition

(with and without). Then, 64 measurements were gathered for each of the two

experimental conditions, and a Wilcoxon statistical test was applied to determine whether

or not there is a significant effect.

Table 51 below show the SASHA-Q for pilot results „with‟ and „without TWs‟.




SASHA Pilots


Without TW With TW4,2

4,4

4,6

4,8

5,0

5,2

5,4

5,6

5,8

6,0

6,2


With-TW / Without-TW No impact P=0.2425

Table 51: SASHA-Q Results for the pilots


conditions. TW management does not impact the pilots' situation awareness. The median

and maximum value is higher in „without‟ conditions than in „with‟ conditions, although

these differences are not significant.

The level of situation awareness is high whatever the TW conditions because the median

values are 5.35 and 5.50 while the maximum of the rating scale is 6.


The pilots' impression is that situation awareness is not impacted by TW use. They rate

level 4 on a five-point scale the fact that the situation awareness is not decreased by the

TWs. The understanding of the TW data displayed on the Navigation Display is easy

although some improvements have to be done in order to have a better anticipation of

the aircraft trajectory on the TW management.




5.2.2.2.3 Situation Awareness Conclusion

TWs use does not impact the pilots' situation awareness. TW data are easy to perceive

and understand. The next TW does, however, need to be displayed in order to improve

pilots' anticipation.

5.2.2.3 Working methods

5.2.2.3.1 Collective aspects between controllers and pilots

This part is analysed in the Controllers' human performance assessment in 5.2.1.4.6.

5.2.2.3.2 Collaboration between captain and first officer

The flight simulation platform was a pilot one-seat. Obviously with such a simulation

environment, it was not possible to observe the relationships between the captain and

the first officer (which is the composition of practically all modern aircraft aircrews).

Because this issue was important for TW assessment, the decision was taken that the

pilot subjects should be experienced captains. The pilot subjects were therefore asked to

consider how TW management could impact the relationships between the captain and

the first officer in terms of their experience of the HIL2 experiment exercises.

The collaboration issues were assessed via the exercise debriefings and during post–run

questionnaires.

The pilots feel that TW management would require more communications between the

captain and the first officer. They rate this impression at level 4 on a five-point scale.

They feel the same regarding the workload, which would increase between captain and

first officer, in order to manage the TWs, share data and have a proper understanding of

the impact on the flight.

As regards the common situation awareness of the aircrew, pilots say that TW

management cannot decrease the aircrew situation awareness. They rate this opinion at

level 4 on a five-point scale.

One interesting issue is to determine which pilot of the aircrew has the best function for

managing the Target Windows. Clearly, the pilots say that the Pilot Flying (PF) will be in

the best position in the cockpit to manage the TWs. This is easily understandable in

terms of the PF tasks, which are to fly the aircraft and manage the trajectory. However,

the Pilot Non Flying (PNF) cannot be excluded from this management process because

they are in charge of communications with ATC. Also, they support the PF and are able to

replace them at any time.

These results show that TWs will impact slightly the collaborative work in the cockpit

between the captain and the first officer, because the workload may be increased but the

situation awareness will not be impacted. Regarding most of the flight phases, this will

not be a concern, but some experimentations are required to assess such a growth in

high workload phases.

5.2.2.3.3 Flying tasks

A limitation of the simulation was that the pilots were not aware of the traffic volume

because data-link was used by the controllers to communicate with non piloted aircraft,

and because the TCAS data were not consistent with the simulated traffic. These

limitations, which were judged minor in a first stage, were in a later phase evaluated as




being impairing. Pilots were not able to understand controllers' decisions or make some

requests to controllers in relation to the surrounding air traffic. These limitations

impacted the flying task, RT communication and TW management.

During the cruise phase, the impact of TWs is low on the flying tasks. Pilots assess this

impact at level 2 on a five-point scale. Some comments said that in quiet periods of the

flight, the TW management is a source of activity which may be useful for fighting

against drowsiness.

The limitations of the use of TWs are in flight phases with high workload, when an

additional load due to the TW management is not acceptable. However, when the

workload is high, the priority is usually safety, and in case of safety concerns TW

management is not the first priority. At any time, aircrew can make decisions which

achieve the safety goals and disregard TW management.

5.2.2.4 Acceptability and Usability

Unanimously, pilots strongly agreed that TWs are easy to use. They rate this easiness at

level 4 on a five-point scale. This impression was largely expressed during the post-run

debriefings. Pilots quickly became familiarized with the concept, and were autonomous in

managing the TWs in the cockpit.

This impression was reinforced by the fact that pilots found TW management easy to

learn (level 4.5 rating on a five-point scale). They were satisfied with the training and

familiarization programme. The training was evaluated as satisfactory, allowing a good

understanding of the concept and its use.

The high level of usability assessed during the simulation is a positive point of the

concept for its future development. This means that the learning process is not a barrier

for the TW experiments, and for potential TW implementation in the real operational

rooms.

A small number of pilots‟ criticisms were directed at the TW display. Their main remarks

are:

The display of the right level (FL275 and FL345) of superimposed target windows

(horizontal levels of sector boundaries). The logic is different for the pilots than

for the controllers because the use is different.




Figure 9: SUPP TW on Navigation Display

The TW data are displayed on the ND. Pilots would like to study the possibility of

displaying them on the MCDU. The ND limitation is the temporal horizon with the

next TWs. The PF configure the ND with low scales. Then, the displayed range

can be insufficient to display the following TW or the next two TWs depending on

the sector length. Although the next TW information can be displayed on the PNF

ND, with a larger scale, it is not convenient for the PF. The PF manages the

aircraft trajectory and the data should be immediately available. Another display,

which allows the next TWs to be displayed, is required to allow the PF to

maintain good flight anticipation. The MCDU could be this display. The issue is

whether to display only on the MCDU or on the two displays (ND and MCDU),

and whether to give to the pilots the choice of the display.

5.3 System Performance Results

As described in the Validation Strategy [3], four performance areas were investigated

during this experiment: capacity, safety, efficiency, and predictability.

5.3.1 Capacity

This KPA addresses the ability of the ATM System to cope with the air traffic demand

[10]. The sector capacity was an independent variable of the HIL2 experiment.

The choice made in the HIL2 experiment for evaluating the KPA capacity, was to assess

two levels of capacity, and not to progressively assess a capacity growth and identify the




breaking point. This choice was made in order to satisfy all experimental goals regarding

the available resources (time of experimentation).

Two traffic loads or capacity were tested during the simulation exercises:

2008 current capacity

2020 forecast capacity

The expected level of traffic in 2020 was determined by the EUROCONTROL service of

statistics and forecast in Brussels. The expected traffic growth between 2008 and 2020 is

forecast at 40% in the measured area.

The measured scenarios [17] were designed taking into account these values to meet the

HIL2 experiment objectives. Table 52 shows the instantaneous traffic load during the

experimentation scenario in the MI sector. This table indicates the two levels of traffic

loads: 2008 between 16:10 and 16:30, and 2020 expected traffic between 16:35 and

17:00. The instantaneous traffic values are:

2008: from 10 to 20 aircraft;

2020: from 15 to 27 aircraft.

All scenarios were built to address the same level of traffic in the two 2008 and 2020

phases.

Nb

of

Aircra

ft

Simulation Time

Geographic Sector Load

0

5

10

15

20

25

30

16:00 16:10 16:20 16:30 16:40 16:50 17:00 17:10 17:20 17:30

Nb

of

Aircra

ft

Simulation Time

Geographic Sector Load

Geo Sect Load

Geo Sect Load (Avg)

0

5

10

15

20

25

30

16:00 16:10 16:20 16:30 16:40 16:50 17:00 17:10 17:20 17:30

Table 52: Instantaneous Traffic into the MI Sector along an Experimental Run




By choosing two levels of capacity, traffic capacity is an independent variable. The

purpose is to see whether, with the 2020 traffic features, the other dependent variables

(mainly the other three KPAs and the human performance) are not negatively impacted.

In this way, HIL2 results show controllers were able to manage the forecast 2020 traffic

properly.

This outcome had to be considered in relation to the known simulation limitations:

Lower level of stress in simulation environment than in real control room.

No pseudo-pilot for the non-piloted aircraft, and so lack of voice communication

with the aircrews, although all the orders were given by a data-link type system

which simulates the ATCOs‟ actions. The data-link type system can only run from

the controller to the aircrew.

In order to avoid immediate and proper execution of controllers' orders by the

data link, delays were implemented for the acknowledgment of orders, and also

for compliance with the orders. This means that the controller has to ensure that

the aircraft carries out the instruction properly.

5.3.2 Safety

Safety is the major concern of Air Traffic Control. The challenge for the future is to

increase the level of capacity with at least the same level of safety as today.

For this reason, safety cannot be considered independently of capacity. Increasing

capacity without keeping a high level of safety is not acceptable.

As the current level of safety is around 10-6, the probability of occurrence of a hazard is

very low and cannot easily be reproduced in HIL experiments. For this experiment safety

was evaluated by:

Assessment of separation performance by the Separation Performance Tool

(SPT) designed by EUROCONTROL [16]. This methodology was selected because

the lack of validity and sensitivity of STCA counting. The HIL1 experiment

showed that there are lot of false alarms which are properly managed by the

controllers without impairing safety. This means that each STCA detected by the

system needs to be classified in terms of its actual impact on safety. This

process was applied in the HIL1 experiment and only a very small number of

STCAs were true. The number was so low that no statistical analysis could be

carried out. Another approach was required. In addition, the STCA calibration

depends on each ACC. It is difficult to draw general conclusions from the results,

and a generic method is required. For the experiment, the STCA was calibrated

with the parameters used by Rome ACC:

Horizontal separation: 5 NM.

Vertical separation: 900 ft.

Time separation: 80 seconds.

Dedicated questions in post-run questionnaires for controllers and pilots.

5.3.2.1 Separation assurance assessment

Separation performance provides the actual and predicted flight times (in minutes) for

different separation bands for all the flights. The separation is measured before and after

controllers' tactical interventions. The actual separation represents the total flight time




that the aircraft have flown during the exercises. Predicted flight time is based on aircraft

trajectories without controllers' interventions, and the closest separation distances

according to the intended trajectories. Separation above 10 NM and 1000 ft are grouped

together with traffic that did not risk loss of separation. The bands of separation are:

< 1 NM and 800 ft;

1.0-2.5 NM and 800 ft;

2.5-4.0 and 800 ft;

4.0-5.0 and 800 ft;

5.0-7.5 and 1000 ft;

7.5-10.0 and 1000 ft;

<10 NM and 1000 ft.

Loss of separation is when aircraft are < 5 NM and 800 ft.

Due to technical problems, the data from the first three runs were lost. To balance the

amount of data, the three other data sets (representing the same traffic patterns, but

opposite conditions, taking into account the rotation on controllers‟ positions) were

excluded. The analysis is based on data from 10 runs (5 runs with target windows and 5

runs without target windows).

Table 53 to Table 56 show the results of traffic separation performance.


sectors





sectors


sectors





sectors


With-TW / Without-TW No impact p=0.910825

According to Table 53 to Table 56, the main result of the flight time distribution present

in the four experimental conditions is that there is no loss of separation. A high safety

level is maintained whatever the traffic load conditions and/or the TW conditions, no loss

of separation is reported here (<5 NM & 800 ft).

The Wilcoxon test applied on the data also shows that separation was not impacted by

TW implementation (p=0.910825).

The majority of the traffic is maintained at more than 10NM and 1000ft. The controllers

successfully separated the aircraft whatever the experimental conditions (traffic load and

TWs) because the flight time of the actual traffic is greater in the band >10 NM than the

flight time of the predicted traffic.

For the two bands between 5 NM and 10 NM, the flight time of the predicted traffic is

greater than the flight time of the actual traffic. An interpretation of this result is that the

majority of potential conflicts are avoided before these separation limits. Such results are

consistent with the hypothesis that controllers have good anticipation in conflict detection

and solving. This result is found in the four experimental conditions.




The SPT tool also provides some data regarding the way the ATCOs ensure separation. It

gives the number of interventions/orders in terms of the time or distance before a

potential loss of separation (so before a potential conflict).

Table 57 and Table 58 show the effect of TW implementation on this number of

interventions, in terms of the time or distance to a potential conflict.

Target Windows*time to LoS; LS Means

Current effect: F(7, 176)=,04287, p=,99990

Effective hypothesis decomposition

Vertical bars denote 0,95 confidence intervals

Target Windows non active Target Windows active

<0.5 min0.5-1min

1.0-2.02.0 - 3.

3.0-4.04.0 - 5.

5.0 -7.57.5 - 10

time to LoS

-10

-5

0

5

10

15

20

25

30

Nb o

f In

terv

entions

Table 57: Number of interventions before time to LoS




Distance to LoS*Target Windows; LS Means

Current effect: F(5, 180)=,03606, p=,99930

Effective hypothesis decomposition

Vertical bars denote 0,95 confidence intervals

Target Windows non active Target Windows active

<2.5 NM 2.5-4.0 4.0-5.0 5.0-7.5 7.5-10 >10

Distance to LoS

-10

-5

0

5

10

15

20

25

Nb

of

Inte

rve

ntions

Table 58: Number of interventions before distance to LoS

According to Table 57 and Table 58, the main result shown by the distribution of the

number of interventions in the four experimental conditions is that TW implementation

has no impact (p=0.99930 and p=0.9990) on the way the ATCOs manage a potential

conflict resolution.

The same safety level is maintained with the different TW conditions. The ATCOs gave

orders to the aircraft at virtually the same distance and time to the potential conflict,

whatever the TWs condition. These results indicate that TW implementation has no

impact on conflict resolution management. It should be noticed that only the

interventions between 0 min and 10 min before the LoS, as well as between 0 NM and 10

NM before the LoS, were counted here. All the data above these ranges were considered

as normal traffic management, in other words not linked with conflict resolutions.

In conclusion, the results given by the traffic separation assurance assessment are:

TWs do not impact the safety level because there was no loss of separation;

TWs do not impact the controllers' anticipation in managing safety;

high traffic loads do not impact safety when controllers have to manage TWs.




5.3.2.2 Post-run Questionnaire and Debriefing Data

5.3.2.2.1 Controllers' opinion

All controllers reported that their safety perception was not impaired by the use of TWs.

They agree that TWs do not impact safety (either positively or negatively). They have no

difficulty in safely managing the traffic in 2008 traffic load conditions. In 2020 traffic load

conditions, they are able to safely manage the traffic but some of them recognise they

require more attention.

The safety questionnaire assesses safety comfort and the occurrence of hazardous

situations. The results show that there is no significant difference (p<0.05) for safety

comfort and the occurrence of hazardous situations whatever the „with‟ and „without‟ TW

conditions, and whatever the traffic load conditions.

In conclusion, controllers agree that TW use does not impact safety (either positively or

negatively) with the 2020 traffic load level, equally „with‟ and „without TWs‟.

5.3.2.2.2 Pilots' opinion

The pilots report that their safety impression is not impaired by TW use. They agree that

the TWs do not impact safety. They have no difficulty in safely managing the aircraft

trajectory and apply the controllers' orders. The impression of safety is rated 4 on a five-

point scale.

The safety questionnaire assesses safety comfort and the occurrence of hazardous

situation. The results show there is no significant difference (p<0.05) for safety comfort

and the occurrence of hazardous situations whatever the „with‟ and „without‟ TW

conditions.

In conclusion, pilots agree that TW use does not impact safety (either positively or

negatively) with the 2020 traffic load level, equally „with‟ and „without TWs‟.

5.3.2.3 Safety Conclusion

Quantitative data, as well as post-run questionnaire and debriefing data are consistent in

stating that safety was not impacted by TW use for the controllers and for the pilots,

even when the capacity matched the 2020 expected traffic load.

5.3.3 Efficiency

KPA efficiency is defined by SESAR [10] [6] as the measurement of the trajectory flown

against the shared business trajectory, in terms of departure delay, fuel burn and flight

time duration. In other words, for the HIL2 experiment, and in reference to the Validation

Strategy [3], the assessed hypothesis is that the implementation of CoO and the

associated TWs positively affect aircraft throughput in the sectors measured.

Control efficiency is then assessed through three indicators:

Flight duration;

Number of fulfilled TWs;

Fuel burned by the aircraft.




5.3.3.1 Flight Duration

The flight duration is calculated by comparing the time flown by each aircraft into the

sector during the experimental exercises, with a reference time which is the time taken

to fly through the sector for the same aircraft without any ATCO actions (simulator flying

the aircraft, following the flight plan). The flight duration is expressed in terms of a

percentage of these two flight times. The overall flight duration is the average of all

aircraft flight durations.

If the controlled flight duration is the same as the reference flight duration, the ratio

between them is 100. If the controlled flight duration is longer than the reference flight

duration, the flight duration is greater than 100. If the controlled flight duration is shorter

than the reference flight duration, the flight duration is less than 100.

Table 59 and Table 60 show the results of flight durations during the simulation exercises

with and without event. Eight measurements were performed for each condition on each

measured sector (KL: Geneva and MI: Milan). The Wilcoxon statistical test was applied to

determine whether or not there is a significant effect caused by use of the TWs.

Table 59 and Table 60 below show the flight duration results for the EXEs on the KL

(Geneva) and MI (Milan) sectors in the four experimental conditions:



2020 traffic, and


80.0

85.0

90.0

95.0

100.0

105.0

110.0

115.0

120.0

2008 2008-TW 2020 2020-TW

FlightDuration for Pseudo-piloted A/C KL1

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- Impacted --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.014 (z:2.21)

p = 0.201 (z:0.84)

p > 0.250 (z:-0.53)

p = 0.174 (z:-0.95)




Table 59: Flight Duration in KL (Geneva) Sector

80.0

85.0

90.0

95.0

100.0

105.0

110.0

115.0

120.0

2008 2008-TW 2020 2020-TW

FlightDuration for Pseudo-piloted A/C MI1

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008TW/2020TW

-- Impacted --

-- No Impact --

-- No Impact --

-- Impacted --

p = 0.005 (z:2.63)

p > 0.250 (z:0.63)

p = 0.071 (z:1.47)

p = 0.047 (z:1.68)

Table 60: Flight Duration in MI (Milan) Sector

Table 61 and Table 62 show the flight duration results for the EXEs of the KL (Geneva)

and MI (Milan) sectors in 2020 traffic conditions with or without event:

2020 traffic,







80.0

85.0

90.0

95.0

100.0

105.0

110.0

115.0

120.0

2020 2020-TW 2020EV 2020EV-TW

FlightDuration for Pseudo-piloted A/C KL1

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- Impacted --

-- Impacted --

p = 0.201 (z:0.84)

p > 0.250 (z:0.11)

p = 0.002 (z:2.84)

p = 0.018 (z:2.10)

Table 61: Flight Duration in KL (Geneva Sector) with Event conditions




80.0

85.0

90.0

95.0

100.0

105.0

110.0

115.0

120.0

2020 2020-TW 2020EV 2020EV-TW

FlightDuration for Pseudo-piloted A/C MI1

Min 25%75% Max

Median


2020EV/2020EV-TW

2020/2020EV

2020-TW/2020EV-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:0.63)

p = 0.148 (z:1.05)

p > 0.250 (z:0.00)

p > 0.250 (z:0.21)

Table 62: Flight Duration in MI (Milan) Sector with Event conditions

The flight duration in the sectors show the following results:


For the KL (Geneva) sector, there is a significant difference (p<0.05) between „without

TW‟ and „with TW‟ conditions for the 2008 traffic load. For the 2020 traffic conditions,

there is no significant difference between „without TW‟ and „with TW‟ conditions. For the

2008 traffic, the „with TW‟ flight duration is shorter than for the „without TW‟ flight

duration. For the 2020 traffic, there is a similar tendency between „with TW‟ and „without

TW‟ conditions, but the difference is not significant. The traffic load condition does not

impact the flight duration.

For the MI (Milan) sector, the tendency is the same as for the KL sector. The flight

durations „with TW‟ are shorter than „without TW‟ whatever the traffic load conditions.

However, the difference between „with TW‟ and „without TW‟ is significant for the 2008

traffic load while the difference is not significant for 2020 traffic load conditions. The

traffic load has a significant impact on the „with TW‟ flight duration because flight

duration is significantly shorter with 2020 traffic load than with 2008 traffic load.


For the KL (Geneva) sector, there is no significant difference (p<0.05) for the flight

duration between the TW conditions for the „with event‟ and „without event‟ condition.

TWs have no impact on the flight duration. There is significant difference for the „with

event‟ conditions on the flight duration whether „with TW‟ or „without TW‟. The „with

event‟ flight duration is shorter than „without event‟ whatever the TW condition.




For the MI (Milan) sector, there is no significant difference (p<0.05) whatever the TW

conditions and the „with event‟ conditions.

The flight duration study does not show a consistent tendency between the different

experimental situations. However, the results seem to indicate that the flight duration is

shorter with Target Windows in a number of situations while there is no clear tendency in

the opposite way (shorter flight duration without Target Windows).

The median values of flight duration with TW are closer to the 100 value than without

TW, indicating that with the TWs, the aircraft flew closer to the flight plan. This suggests

better adherence to the trajectory. This outcome means that TW use increased the traffic

efficiency.

5.3.3.2 Number of fulfilled TWs

The CoO concept and associated TWs were designed to improve traffic efficiency. Thus,

the number of aircraft fulfilling their TWs is a strong indicator of the concept validity and

traffic efficiency. If controllers are not able to fulfil the TWs and if too many aircraft are

out of their TWs, the concept will lose its relevance and traffic efficiency will decrease.

As observed during the first experiment, the reasons why the controllers could not fulfil

TWs could be various and mainly due to local/sector uncertainties. Weather conditions,

conflict-solving and traffic management have been observed as reasons for modifying the

initial flight plan and flying around the TWs during the experiment.

For each aircraft, the simulator recorded whether or not the aircraft fulfil their exit sector

TW. The number of „out TW‟ aircraft was gathered for each exercise.

Table 63 to Table 66 below show the percentage of „out TW‟ aircraft during the simulation

exercises with and without event. Eight measurements were performed for the 2008 and

2020 „with TW‟ conditions in each measured sector (KL: Geneva and MI: Milan). The

Wilcoxon statistical test was applied to determine whether or not the traffic load level

impacted the number of „out TW‟ aircraft.

Table 63 and Table 64 show the percentage of „out TW‟ aircraft results for the EXEs of

the KL (Geneva) and MI (Milan) sectors in the two experimental conditions:

2008 current traffic with TW, and





0.0

20.0

40.0

60.0

80.0

100.0

2008-TW 2020-TW

Target Windows Out (Percentage) KL1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.0502008TW/2020TW -- No Impact -- p > 0.250 (z:-0.42)

Table 63: Percentage of „out TW‟ Aircraft in KL Sector

0.0

20.0

40.0

60.0

80.0

100.0

2008-TW 2020-TW

Target Windows Out (Percentage) MI1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.0502008TW/2020TW -- No Impact -- p > 0.250 (z:0.42)

Table 64: Percentage of „out TW‟ Aircraft in MI Sector




Table 65 and Table 66 show the percentage of „out TW‟ aircraft results for the EXEs of

the KL (Geneva) and MI (Milan) sectors in 2020 traffic conditions with or without event:

2020 traffic with TW, and


0.0

20.0

40.0

60.0

80.0

100.0

2020-TW 2020EV-TW

Target Windows Out (Percentage) KL1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.0502020-TW/2020EV-TW -- No Impact -- p > 0.250 (z:-0.42)

Table 65: Percentage of „out TW‟ Aircraft in KL Sector with event




0.0

20.0

40.0

60.0

80.0

100.0

2020-TW 2020EV-TW

Target Windows Out (Percentage) MI1

Min 25%75% Max

Median

Wilcoxon test: Variable impact results if p < 0.0502020-TW/2020EV-TW -- No Impact -- p = 0.201 (z:-0.84)

Table 66: Percentage of „out TW‟ Aircraft in MI Sector with event

The percentage of „out TW‟ aircraft in the sectors show the following results:


There is no significant difference (p<0.05) between the two traffic load conditions


The median value is always equal to 0 whatever the sectors and the traffic load.

These results show that the percentage of Target Windows Out is very low and fully

compatible with operational use. The renegotiation process has to be initiated very

rarely.


As for the „without event‟ conditions, there is no significant difference (p<0.05) between

the two traffic load conditions whatever the controlled sector.

Also, the median values are always equal to 0 whatever the sectors and the traffic load.

However, there is a tendency to have more Target Windows Out with the event. This

result is not surprising because avoidance of the event means moving away from the

planned route and sometimes not fulfilling the TW.

Although the number of TWs out is a little bit higher than for the „without event‟

conditions, this number is fully compatible with an operational use.




The conclusion of these results is that the percentage of „out TW‟ is relatively low and

appears to be acceptable. TW fulfilment was not statistically impacted by TW use, the

traffic loads, or an event. However, TW fulfilment seems to be sensitive to the sector

shape, airspace structure, and traffic conditions.

During the debriefing sessions, the controllers confirmed that they were not disturbed by

the number of TWs Out, thinking it was easy to manage them through an RT negotiation

with the following ANSP. The number of „out TW‟ aircraft was found to be acceptable by

all controllers.

5.3.3.2.1 Fuel burned by the aircraft

For the assessment of fuel consumption, the model available in the flight simulator was

used. The idea of analysing the fuel burned by the aircraft is an attempt to evaluate one

factor of the flight cost. If differences for fuel burned during the sector crossing are

observed, it will be possible to extend such results to the air transport system with

modelling tools.

The calculation of the fuel burned is made by comparing the flight simulator fuel values

at the sector entrance and exit.

Eight measurements were gathered for each of the four experimental conditions (2008

and 2020 traffic loads „with‟ and „without TWs‟), and a Wilcoxon statistical test was

applied to determine whether or not there is a significant effect.

Table 67 and Table 68 show the fuel burn results for the aircraft piloted on each flight

simulator platform in the four experimental conditions:



2020 traffic, and





0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

1000.0

2008 2008-TW 2020 2020-TW

Fuel burned FS1_PIL

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008-TW/2020-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p = 0.248 (z:0.68)

p > 0.250 (z:-0.11)

p > 0.250 (z:-0.05)

p > 0.250 (z:-0.42)

Table 67: Fuel burned Cockpit1

0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

1000.0

2008 2008-TW 2020 2020-TW

Fuel burned FS2_PIL

Min 25%75% Max

Median


2020/2020-TW

2008/2020

2008-TW/2020-TW

-- No Impact --

-- No Impact --

-- No Impact --

-- No Impact --

p > 0.250 (z:-0.53)

p > 0.250 (z:-0.58)

p > 0.250 (z:-0.21)

p > 0.250 (z:0.11)

Table 68: Fuel Burned Cockpit 2




The results show that there is no significant difference (p<0.05) for the fuel burned by

the aircraft whatever the traffic load conditions and whether or not TWs are managed.

The Target Windows, as well as the traffic load, do not impact the fuel burned by the

aircraft.

This assessment of this cost factor is the first attempt made with the tools available for

the HIL2 experiment. The model of fuel consumption is that of the simulator. The pilots

were asked to give their impressions about consumption, and they agree with the flight

simulator values. The values are consistent with true aircraft consumption values.

The lack of difference in the fuel consumption is a strong point as regards the efficiency

of the TW situations.

5.3.3.3 Efficiency Conclusion

The data collected during the simulation exercises allow us to conclude that TWs do not

impact traffic efficiency. On the contrary, and in an appropriate sector, where the sector

shape, size and airspace may positively impact the flight duration, increased traffic

efficiency could be observed.

Such outcomes confirm that TWs may actively contribute to improving traffic efficiency,

mainly by ATCOs managing the traffic closer to the schedule. The expected benefits need

to be balanced in the future with the impact of TWs not fulfilled, with the renegotiation

process and new contracts of objectives. The HIL3 experiment will assess the

renegotiation process.

In this experiment, the number of „out TWs‟ aircraft was found to be acceptable by all

controllers. Controllers were not perturbed by these „out TW‟ aircraft – they believe that

it will be easy to manage them. However, this should be verified in the next experiment.

The analysis of the piloted aircraft fuel burn shows that the TWs do not increase the flight

cost. Despite the limitations of the aircraft simulation model, this result is a first attempt

in a Human in the Loop experiment to assess the cost impact of a new ATM concept.

5.3.4 Predictability

Predictability indicators are defined by SESAR [10] [6] as the measurement of the flown

trajectory against the reference business trajectory. This is virtually the same in our

experiment as indicators are used for assessing efficiency. Indeed in our experiment, the

hypothesis regarding the predictability issue is that the implementation of the CoO and

associated TWs will ensure that the schedule is respected.

This means that we have reached the same conclusion for traffic predictability outcomes

as for efficiency outcomes, namely that the TW did not impair traffic predictability, and

may even improve it with an appropriate control sector shape, size and airspace

structure.




6 Conclusion

The aim of this simulation was to present the CoO concept and associated TWs to

controllers and pilots, to investigate the impact of this concept on their activity and

relationships, and to evaluate the operational acceptability from a controller‟s and pilot's

point of view.

To summarize, controllers and pilots are very positive about the concept. However, they

believe that the potential benefits are more pertinent to airlines and the air transport

system as a whole than to controllers and pilots themselves. The HIL2 experiment results

are consistent with the HIL1 experiment as far as the controller issues are concerned.

The results are valuable in achieving a better understanding of the challenges raised by

the implementation of the Contract of Objectives in Air Traffic Control for the ANSPs,

controllers and airlines pilots. The HIL2 experiment outcomes have to be interpreted in

the light of the limitations of the simulation devices.

6.1 Human Performances Results

6.1.1 Controllers

Results suggest that the Contract of Objectives concept is manageable with the current

2008 and expected 2020 traffic loads in the two measured sectors, without any impact

on traffic safety. Controllers considered TW management to be feasible and acceptable,

although TWs add certain constraints when considering conflict resolution. The usability

and ease of management of TWs are similar for the executive and PLNs, although TWs

are more suitable for the PLN than for the executive controller.

Controllers are more constrained by the heavy traffic load than by TW use. However, TW

management involves more information, which increases the perception of workload. But

this increase in information is also considered to be a positive aspect for improving

situation awareness. Objectively, quantitative data reveal that TW use has no impact on

workload and situation awareness. This outcome is a strong indicator for future

development and concept acceptability.

Fundamentally, the Contract of Objectives does not modify the way the controllers work.

The communication and cooperation processes are not impacted, and the common

situation awareness between executive and PLN is assessed as being satisfactory. Some

slight changes are described for the executive when he (she) has to solve conflicts. He

(she) also has to analyse the TW data to find the best solution for conflict resolution. The

PLN is not impacted. Controllers also say that it is easy to bear in mind safety as the first

priority. Thus, mainly in 2020 conditions, they concentrate on conflict solving, returning

subsequently to fulfilment of the TWs. There are no difficulties associated with the

achievement of safety.

6.1.2 Pilots

Results suggest that the Contract of Objectives concept is manageable in the cockpit by

the aircrew without any impact on safety. Pilots judged TW management to be feasible

and acceptable, although TWs add certain management constraints. From the pilot's

point of view, the management of TWs in the flying task has to be the responsibility of

the Pilot Flying (PF). However, the Pilot Not flying (PNF) needs to be kept in the loop of

TW management.




Pilots are slightly constrained by TW use, which modifies the workload and increases

collaboration between the crew members. They have the impression that workload is

increased and the quantitative data confirm this feeling. However, the level of workload

remains low. It is not a concern for the flying phases tested during the experiment, but

the issue has to be considered for heavy workload phases or emergency situations.

However, pilots are aware that safety is always the first goal, and they are able to give

up the TW constraint. Like for the controllers, the TWs are also considered as positive for

improving situation awareness. Objectively, quantitative data reveal that TW use has no

impact on situation awareness. These outcomes are strong indicators for the future

development and the concept acceptability.

Fundamentally, the Contract of Objectives does not modify the way the pilots work. The

communication and cooperation processes with controllers are not impacted and the

common situation awareness with controllers is assessed as being improved. Cooperation

between the two crew members is thought to be easy to manage with TWs, although

pilots feel this will require a little bit more work.

Pilots stress that the rules have to very clear regarding the approach of the aircrew to

TW management in the case of heavy workload in the cockpit. They stress that safety is

always the first priority. The relationship with the controllers also has to be defined

clearly in terms of responsibility for deciding whether or not to apply TWs regarding the

flight or traffic management constraints.

6.2 System Performance Results

The results obtained in terms of System Performances indicated that the capacity

expected in 2020 was properly and safely managed. The Efficiency and Predictability

KPAs were assessed quantitatively and qualitatively. Data showed clearly they were not

impacted by TW management. These KPAs were slightly improved in some cases, such as

in the fields of predictability and efficiency, although the improvements were not

statistically significant.

Safety quantitative data (obtained through the use of SPT), as well as qualitative data

obtained through post-run questionnaires and debriefings, are consistent in confirming

that safety was not impacted by TW use for the controllers and for the pilots, even when

the capacity matched the 2020 expected traffic load.

6.3 Lessons Learned

6.3.1 Training

Training is an important issue when introducing a new concept.

The controllers and pilots judge the concept to be usable and easy to learn and the

simulation devices and training and familiarization programmes to be suitable.

Training sessions as well as the operators‟ handbook and the presentations given in-situ

provided:

sufficient knowledge of the concept;

sufficient knowledge of the platform, HMI and supporting tools;




operators with sufficient familiarisation with the airspace, procedures and

working methods.

Controllers and pilots say further supplementary training will not be required for the next

HIL experiment.

Nevertheless, we should reconsider how to position this concept in the future 2020

environment and in relation to with the SESAR program. The simulation showed that

controllers had some difficulties envisaging all the consequences the Contract of

Objectives may have on the Air Traffic Management system. Most of the time, they still

assessed TWs from a sector point of view, not considering all the consequences on the

air transportation system. They tried to envisage the Contract of Objectives through the

current constraints and the same type of relations the actors of air transportation system

currently have. This justifies an additional effort to clarify the impact of TW use on

relationships with other air traffic control partners and to be ensure that this

development is properly understood by the subject controllers.

For the pilots, understanding all the consequences the Contract of Objectives may have

on the Air Traffic Management system is easier. Although they are aware that the

concept will change the current pilots' mentality regarding the way to manage the

schedule and cost constraints, they are clearly in favour of the purpose - to optimise the

airline business. They found the concept useful in that TWS give them a better awareness

of the traffic constraints.

6.3.2 Platform, HMI and Recordings

During this experiment, controllers stressed the need to pay more attention to the HMI

by having enhanced integration of the Target Windows data with the aircraft flight plan

data. They greatly appreciate the specific collaborative tools designed for this simulation

and the SkyGuide tools (Medium Term Conflict Detection). Controllers agree the tools are

essential for coping with traffic growth. The platform was judged acceptable by the

controllers to assess such concepts, although some improvements, such as label data,

should be considered.

Pilots considered the flight simulation platform to be satisfactory. The platform allows

them to have a good representation of the concept and its impacts on the pilots' tasks.

However, some limitations were pointed out, and more complete and exhaustive

evaluations require a more complex flight simulator. Efforts have concentrated more on

air environment fidelity and onboard systems than on a two-seat simulator.

Recordings should also be improved regarding the communication between controllers

and between controllers and pilots in order to have more accurate analysis of

collaboration processes.

6.3.3 TW Modelling

Some improvements in the calculation of TWs are reported as a real need, mainly in

relation to the superimposed TWs. In addition, controllers and pilots think a high level of

acceptability of TWs also requires a good understanding of the basic principles of their

calculation.




6.3.4 Working Methods

TW management is an additional factor that has to be considered when executive and

PLNs analyze potential conflict resolutions. The results of this experiment showed that

this added constraint slightly impacted the way the EXEs solved the conflict. This should

be further studied, mainly by evaluating the impact of renegotiation and new contracts.

TWs slightly impact the way the pilots work. TWs are additional information to take into

account, which can monopolize some mental resources for its management in the cockpit

and with the controller. However, the strain is low and TWs are never a priority in heavy

workload periods.

6.4 Future Studies

This experiment was the second step of the operational assessment planned to validate

the CoO concept. This will be followed by the third and last step in June 2010, dealing

with the renegotiation process and its impact on the aircrews and controllers. This study

will be very important for the evaluation of the acceptability of the concept to Air

Transportation System actors, and in particular it will assess TW recalculation in

operational time horizons.




7 References

The available documentation is described hereafter:

[1] EC CATS Contract TREN/07/FP6AE/S07.75348/036889

[2] CATS Description of Work (Annex 1 of [1])

[3] CATS Validation Strategy D1.3.1 v2.1

[4] CATS Concept of Operations Definition D1.2.1 v1.0

[5] SESAR Definition Phase-Deliverable D1- Air Transport Framework - The current

situation

[6] SESAR Definition Phase – D2 Air Transport Framework - The performance target

[7] SESAR Definition Phase-Deliverable D3- The ATM Target Concept

[8] SESAR Definition Phase-Deliverable D4- ATM Deployment Sequence

[9] European Operational Concept Validation Methodology - Version 2

[10] EP3 Performance Framework E3-WP2-D2.0-03-RQT

[11] CATS HIL1 Operational Scenario Description D1.3.3 v1.0

[12] Endsley, M. R.,1995, Toward a theory of situation awareness in dynamic

systems. Human Factors 37, 32-64

[13] Hart, Sandra G., Lovell E Staveland, 1988, Development of NASA TLX, in Human

Mental Workload , by Hancock, P.A. and Meshkati, N., Plenum, NY, Elsevier

[14] EUROCONTROL, 2003, Solutions for Human-Automation Partnerships in

European ATM, the development of Situation awareness Measures in ATM

systems, Document HRS/HSP 005 REP 01, Brussels, Belgium, EUROCONTROL

[15] Hollander, M. and Wolfe, D. A., 1999, Nonparametric Statistical Methods (2nd

Ed.)

[16] Pinska, E.; Hickling, B. (2009) Measuring En Route Separation Assurance

Performance. In proceedings of EUROCONTROL Safety R&D Seminar 2009,

Munich, Germany

[17] CATS HIL2 Experimental Scenario Description D1.3.4 v1.0

[18] CATS HIL2 platform and environment Description D2.1.4.2 v1.0




8 Annexes

The pages below present the different questionnaires used during the experiment.

8.1 Over The Shoulders Observations:

Run:

Observer: EXE: IM VT RL LX

Sector: MI1 KL1 PLN: IM VT RL LX

Maintaining separations :

- Checks separation and evaluate traffic

movement to ensure separation standards are

maintained

- Detects and resolves impeding conflictions

- Analyzes flight plans and issues clearances

- Considers aircraft performance parameters

when issuing clearances

- Establishes and maintains proper aircraft

identification

- Properly uses separation procedures to ensure

safety

Maintaining efficient air traffic

flow :

- Accurately predicts sector traffic overload and

takes appropriate action

- Ensure clearances require minimum flight

path changes (collected data)

- Controls traffic in a manner that ensures

efficient and timely traffic flow

- When necessary, issues a new clearance to

expedite traffic flow

- Reacts to / Resolves potential conflictions

efficiently

Maintaining attention & SA :

- Maintains awareness of total traffic situation

- Assigns requested altitude in timely manner




- Reviews and ensure appropriate route of flight

- Scans properly for air traffic events,

situations, potential problems, etc.

- Remembers, keeps track of, locates, and if

necessary orients aircraft

- Descends arrivals in timely manner

- Accepts / Performs timely hand offs

Coordinating :

- Performs handoff and pointout procedures correctly

- Effectively coordinates clearances, changes in aircraft destinations, altitudes, etc.

- Provides complete / accurate position relief briefings

- Performs required coordinations effectively - Initiates and receives handoffs and pointouts in an

efficient and effective manner

Communicating – Cooperation :

- Common traffic picture

- Reply to requests (questions, data, etc.) - Support to decision makings - Proper task sharing

Performing multiple tasks :

- Shift attention between several aircraft when

necessary

- Keep track of a large number of aircraft /

events at a time

- Prioritizes activities effectively

- Communicates in a timely fashion while

performing other actions

- Returns to what he / she was doing after an

interruption

Managing sector workload :

- Handles heavy, emergency, and unusual

traffic situations effectively

- Stays calm, focused, and functional in busy

and stressful conditions

- Handles unexpected situations effectively

(weather, etc.)

- Uses contingency or 'fall-back" strategies

effectively

Overall performance




8.2 Post-Experimental Questionnaire

8.2.1 Pilots

POST-EXPERIMENTAL QUESTIONNAIRE

Pilots

Your opinion is very important for the evaluation of the effectiveness of the Contract of

Objectives / Target Windows concepts. Consequently, we would ask you to answer the

enclosed questions giving your individual opinion and personal experience with Contract

of Objectives / Target Windows concepts.

All the data collected during the HIL2 experiment will be processed in strict confidence.

Only the analysis team of the experiment will see your questionnaire. They will not pass

any personal details and opinions to anyone outside the team

Instructions

The questionnaire contains a number of statements on aspects of the pilot tasks you

performed. Please make your opinion on how much you agree or disagree with each

statement, by making a cross in the box that comes closest to your opinion, as shown

below

Example : Towers should be built even higher to give a better

view to the controllers

Strongly

disagree Disagree Undecided Agree Strongly

agree

X

The cross mark means that you "strongly agree" with the idea

that towers should be built even higher

Please answer all the items quickly in order that they are given but do not cross check

your answers to previous items. Please put any comments to explain your decisions

further in the free spaces below the items (overleaf with reference to question number if

necessary). Please work on your own – do not discuss any questions with your colleagues

while filling in the questionnaire.

Name: CR EM




A. PILOTS' TASK

A-1. Target Windows are easy to use

Strongly


agree

A-2. Target Windows make easier the trajectory management

Strongly


agree

A-3. Target Windows do not impact the pilot’ activity

Strongly


agree

A-4. Target Windows management requires time and mental resources

Strongly


agree

A-5. Target Windows increase the trajectory predictability




Strongly


agree

A-6. Target Windows increase the Pilot's workload

Strongly


agree

A-7. Target Windows decrease the Pilot's situation awareness

Strongly


agree

A-8. Target Windows management could impact tasks achieving by pilots in the

cockpit

Strongly


agree

If yes, what tasks may be impacted?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

A-9. Target Windows management may have adverse impacts on safety




Strongly


agree

If yes, what impacts?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------




C. COMMUNICATION / COOPERATION

Between pilots and controllers

C-1. Target Windows require having more communication with the controller

Strongly


agree

C-2. Phraseology is found easily in order to communicate about Target Windows

with the controller

Strongly


agree

C-3. Target Windows facilitate cooperation with the controller

Strongly


agree

C-4. Target Windows increase the workload to cooperate with the controller

Strongly


agree

C-5. Target Windows allow for representing better the controllers' intents about

aircraft trajectory management




Strongly


agree

C-6. Target Windows communications between the controller and the aircrew

may be done by data-link

Strongly


agree

Between the Captain and the First Officer

The relationships into the cockpit between the Captain and the First Officer were not

simulated, but could you transpose your flying experience about theses relationships to

the target windows management?

C-7. Target Windows will require more communication between the captain and

the first officer

Strongly


agree

C-8. Target windows may decrease the situation awareness between the two

crewmembers

Strongly


agree

C-9. Target Windows may increase the workload between the Captain and the

First Officer




Strongly


agree

C-10. Target Windows is mainly managing by the Pilot Flying

Strongly


agree

C-11. Target Windows may be managing by the Pilot Non Flying

Strongly


agree

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------




D. GENERAL

D-1. Target Windows is a promising concept in order to cope with the future

European ATM challenges

Strongly


agree

D-2. Target Windows concept is easy to use

Strongly


agree

If necessary, what improvements are required?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

D-3. Target Windows are systematically used for managing the trajectory

Strongly


agree


----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

D-4. Target Windows facilitate the work with controllers for managing aircraft

trajectory




Strongly


agree

D-5. Target Windows do not decrease the aircrew performance

Strongly


agree

D-6. It is easy to learn how to use the Target Windows

Strongly


agree

D-7. Target Windows display is easy to use

Strongly


agree


----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------




D-8. More tools (warnings, communication, etc.) are needed for using Target

Windows and managing properly the aircraft trajectory

Strongly


agree

What tools would you like to have?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------




E. ABOUT THE SIMULATION

E-1. The simulation enabled me to obtain a good understanding of interactions

between controllers and aircrews

Strongly


agree

E-2. The Flight simulator enabled me to obtain a good understanding of Target

Windows impacts on the Pilot's activity

Strongly


agree

What improvements are required for a next Human In the Loop Experiment?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------

----------------------------------------------------------------------------------------------

-------------

E-3. The Flight simulator enabled me to obtain a good understanding of impacts

of Target windows on aircrew' activity.

Strongly


agree

What improvements are required for a next Human In the Loop Experiment?




----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------

----------------------------------------------------------------------------------------------

-------------

E-4 for better evaluating the impacts of Target windows on aircrew's activity, a

two-seat flight simulator is absolutely required.

Strongly


agree

E-4. The training and familiarization received prior to the measured exercises

were sufficient

Strongly


agree

E-5. The fact that I am not very familiar with the HMI made it difficult for me to

judge the Contract of Objectives concept

Strongly


agree

E-6. The fact that I am not familiar with the Contract of Objectives (Target

Windows) concept made it difficult for me to judge it




Strongly


agree

E-7. The piloted flights used were appropriate for testing the interactions

between controllers and pilots

Strongly


agree

THANK YOU VERY MUCH FOR YOUR

CO-OPERATION AND CONTRIBUTION

8.2.2 Controllers

POST-EXPERIMENTAL QUESTIONNAIRE

Air Traffic Controllers

Your opinion is very important for the evaluation of the effectiveness of the Contract of

Objectives / Target Windows concepts. Consequently, we would ask you to answer the

enclosed questions giving your individual opinion and personal experience with Contract

of Objectives / Target Windows concepts.

All the data collected during the HIL2 experiment will be processed in strict confidence.

Only the analysis team of the experiment will see your questionnaire. They will not pass

any personal details and opinions to anyone outside the team

Instructions




The questionnaire contains a number of statements on aspects of the ATC tasks you

performed. Please make your opinion on how much you agree or disagree with each

statement, by making a cross in the box that comes closest to your opinion, as shown

below

Example : Towers should be built even higher to give a better

view to the controllers

Strongly


agree

X

The cross mark means that you "strongly agree" with the idea

that towers should be built even higher

Please answer all the items quickly in order that they are given but do not cross check

your answers to previous items. Please put any comments to explain your decisions

further in the free spaces below the items (overleaf with reference to question number if

necessary). Please work on your own – do not discuss any questions with your colleagues

while filling in the questionnaire.

Name: IM VT RL LX




A. EXE

A-1. Target Windows are easy to use for the EXE

Strongly


agree

A-2. Target Windows make easier the traffic management for the EXE

Strongly


agree

A-3. Target Windows increase the traffic safety the EXE has to manage

Strongly


agree

A-4. Target Windows modify the way the EXE detect potential conflicts

Strongly


agree

A-5. Target Windows modify the way the EXE resolve conflicts




Strongly


agree

A-6. Target Windows increase the traffic capacity the EXE has to manage

Strongly


agree

A-7. Target Windows increase the traffic efficiency the EXE has to manage

Strongly


agree

A-8. Target Windows increase the traffic predictability the EXE has to manage

Strongly


agree

A-9. Target Windows increase the EXE workload

Strongly


agree

A-10. Target Windows decrease the EXE situation awareness




Strongly


agree

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------




B. PLN

B-1. Target Windows are easy to use for the PLN

Strongly


agree

B-2. Target Windows make easier the traffic management for the PLN

Strongly


agree

B-3. Target Windows increase the traffic safety the PLN has to manage

Strongly


agree

B-4. Target Windows modify the way the PLN detect potential conflicts

Strongly


agree

B-5. Target Windows modify the way the PLN manage the traffic




Strongly


agree

B-6. Target Windows increase the traffic capacity the PLN has to manage

Strongly


agree

B-7. Target Windows increase the traffic efficiency the PLN has to manage

Strongly


agree

B-8. Target Windows increase the traffic predictability the PLN has to manage

Strongly


agree

B-9. Target Windows increase the PLN workload

Strongly


agree

B-10. Target Windows decrease the PLN situation awareness




Strongly


agree

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------




C. COMMUNICATION / COOPERATION

Between controllers

C-1. Target Windows require having more communication between the

Executive and the PLNs

Strongly


agree

C-2. Target Windows decrease the cooperation work between the Executive and

the PLNs

Strongly


agree

C-3. Target Windows increase the workload to cooperate between the Executive

and the PLNs

Strongly


agree

C-4. Target Windows decrease the common picture of the traffic between the

Executive and the PLNs

Strongly


agree

C-5. Target Windows facilitate cooperation with adjacent sectors




Strongly


agree

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------------------------

----------------------------------------------------------------------------------------------

-------------------

Controllers – Pilots

C-6. Target Windows require having more communication between the

Executive and the Pilots

Strongly


agree

C-7. Vocabulary is found easily in order to communicate about Target Windows

with the pilots

Strongly


agree

C-8. Target Windows facilitate cooperation between the Executive and the Pilots




Strongly


agree

C-9. Target Windows increase the workload to cooperate between the Executive

and the Pilots

Strongly


agree

C-10. Target Windows allow for representing better the pilots' intents and

aircraft trajectory

Strongly


agree

C-11. Target Windows communications between the Executive and the aircrew

may be done by data-link

Strongly


agree

Comments:

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

--------------------------------------

----------------------------------------------------------------------------------------------

-------------------




D. GENERAL

D-1. Target Windows is a promising concept in order to cope with the future

European ATM challenges

Strongly


agree

D-2. Target Windows concept is easy to use

Strongly


agree


----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

D-3. Target Windows are systematically used for building the traffic picture

Strongly


agree


----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------




D-4. Target Windows facilitate the work with pilots for managing aircraft

trajectory

Strongly


agree

D-5. Target Windows constraint the way you maintain separation between

aircraft

Strongly


agree

D-6. Target Windows do not decrease your performance

Strongly


agree

D-7. It is easy to learn how to use the Target Windows

Strongly


agree

D-8. Target Windows display is easy to use




Strongly


agree


----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------

D-9. More tools are needed for using Target Windows and managing properly

the traffic with them

Strongly


agree

What tools would you like to have?

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------

---------------------------------------------------------




E. ABOUT THE SIMULATION


between ANSPs due to the Contract of Objectives

Strongly


agree


between controllers and the pilots

Strongly


agree

E-3. The simulation enabled me to obtain a good understanding of Target

windows impacts on the Controller's activity

Strongly


agree

E-4. The training and familiarization received prior to the measured exercises

were sufficient

Strongly


agree




E-5. The fact that I am not familiar with the HMI made it difficult for me to

judge the Contract of Objectives concept

Strongly


agree

E-6. The fact that I am not familiar with the Contract of Objectives concept

made it difficult for me to judge it

Strongly


agree

E-7. The traffic samples used were appropriate for testing the interactions

between ANSPs due to the Contract of Objectives

Strongly


agree

E-8. The piloted flights used were appropriate for testing the interactions

between controllers and pilots

Strongly


agree




THANK YOU VERY MUCH FOR YOUR

CO-OPERATION AND CONTRIBUTION




8.3 Safety Questionnaire

Run: Name: IM VT RL LX

Sector: MI1 KL1 Control

Position: EXE PLN

OVERALL SUBJECTIVE RISK LEVEL

Please rate your safety comfort

□ LOW □ MODERATE

□ HIGH

Did you experience any situation that could have become

hazardous?

□ None

□ Maybe one situation that could have developed

□ One situation that could have lead to an accident

□ Definitively a hazardous situation

How do you rate the following contributing factors to the

incident?

Simulation environment _______ % Man machine interface

_______ %

TW management __________ %




ONLY IF AN EVENT HAS BEEN ENCOUNTERED

How serious was the situation:

□ Minor convenience (with slight reduction in safety margins and slight

increase in workload)

□ Serious disturbance (significant workload increase or SA degradation

without reduction of safety margins)

□ Major disturbance (significant reduction of safety margins, significant

workload increase)

□ Hazardous incident (large disturbance in safety margins, stress and

adverse effects)

□ Catastrophic incident (total loss of safety margins, total inability to provide

safe ATM services)

How do you think this situation could be avoided?

………………………………………………………………………………………………….

………………………………………………………………………………………………….

………………………………………………………………………………………………….

……………………………………………………………………………………………….....




8.4 Situation awareness for SHAPE (SASHA) questionnaire

8.4.1 Controllers

This questionnaire is designed to assess you level of situation awareness during the

previous working period. Please read carefully through the list of statements and select

the rating that most accurately reflects your thoughts and activities. For each statement,

there are seven possible ratings. Make your selection by marking the number

corresponding to the appropriate rating.

Please do not leave any statement blank. If you‟re unsure which rating to choose, select

the one that fits best. Complete the questionnaire from the viewpoint of the controller

role you held in the previous working period.

Run: Name: IM VT RL LX

Sector: MI1 KL1 Control Position: EXE PLN

In the previous working period(s)..




8.4.2 Pilots:

1) ... J'anticipe ce qui va se passer 0 1 2 3 4 5 6

Jam

ais

Rar

emen

tQ

uelq

uefo

is

Ass

ez s

ouvent

Sou

vent

Très

sou

vent

Touj

ours

2) ... Je me focalise sur un point

particulier lié à la réalisation du vol 0 1 2 3 4 5 6

3) ... Je peux oublier quelque chose

d'important 0 1 2 3 4 5 6

4) ... Je peux planifier et organiser mon

activité comme je veux 0 1 2 3 4 5 6

5) ... J'ai été supris par quelque chose

que je n'attendais pas 0 1 2 3 4 5 6

6) ... Certaines informations me

manquaient et j'ai du les chercher0 1 2 3 4 5 6

Thank you for completing this questionnaire

Indicatif de l'avion :

HIL1 – October 2008




8.5 Instantaneous Self Assessment (ISA)

Subject were prompted every 5 minutes to make a rating among 5 levels of workload,

from Very Low to Very High

report d2.1.2 hil2 primary results analysis... project co-funded by the european community within...

Documents