title of the documentihmaero.recherche.enac.fr/data/documents/e.02-24... · d2.2 – functional...

Functional HMI1

Document information

Project Title MoTa

Project Number E02.24

Project Manager ENAC

Deliverable Name Functional HMI1

Deliverable ID D2.2

Edition 01.01.00

Template Version 03.00.00

Task contributors

ENAC, ISAE..

Abstract

This document describes the design of a ground controller position that facilitates theinput of a variety of clearances and the collaboration with autonomous vehicles.

Project Number E02.24 Edition 01.01.00D2.2 – Functional HMI1

Authoring & ApprovalPrepared By - Authors of the document.

Name & Company Position & Title Date

F.ANDRE, ENAC Research engineer 27/08/2015

M.COUSY, ENAC Research engineer

Reviewed By - Reviewers internal to the project.


Z.CHUA, ISAE-Supaero Post Doctorate 27/08/2015

R.BENHACENE, ENAC Head of IHMA

Reviewed By - Other SESAR projects, Airspace Users, staff association, military, Industrial Support, other organisations.


<Name / Company> <Position / Title> <DD/MM/YYYY>

Approved for submission to the SJU By - Representatives of the company involved in the project.


R.BENHACENE, ENAC Head of IHMA 21/09/15

Rejected By - Representatives of the company involved in the project.


<Name / Company> <Position / Title> <DD/MM/YYYY>

Rational for rejection

None.

Document History

Edition Date Status Author Justification

00.00.01 27/08/2015 Draft M.Cousy New Document

01.00.00 27/08/2015 Release M.CousyCorrection after internalreview.

01.01.00 21/09/2015 Release M.Cousy Correction after SJU review

Intellectual Property Rights (foreground)

This deliverable consists of SJU foreground.

2 of 30

©SESAR JOINT UNDERTAKING, 2011. Created by [Member(s)] for the SESAR Joint Undertaking within the frame of theSESAR Programme co-financed by the EU and EUROCONTROL. Reprint with approval of publisher and the source properly

acknowledged.


Table of Contents

1 INTRODUCTION..........................................................................................................................................5

1.1 PURPOSE OF THE DOCUMENT...................................................................................................................51.2 ACRONYMS AND TERMINOLOGY.............................................................................................................5

2 OVERALL TECHNICAL CONSIDERATIONS........................................................................................6

2.1 HARDWARE CHOICES...............................................................................................................................62.2 DJNN INTERACTIONS LIB..........................................................................................................................6

3 CONTEXT......................................................................................................................................................7

3.1 WORKING ENVIRONMENT........................................................................................................................73.2 ARRIVAL SCENARIO...............................................................................................................................103.3 DEPARTURE SCENARIO...........................................................................................................................113.4 INCIDENT SCENARIO...............................................................................................................................113.5 TAXIBOTS SCENARIO..............................................................................................................................11

4 DESIGN METHODOLOGY......................................................................................................................13

5 FEATURES DESIGN..................................................................................................................................14

5.1 INFORMATION VISUALISATION...............................................................................................................145.1.1 Interactive radar image...................................................................................................................145.1.2 Paper strips replacement.................................................................................................................155.1.3 System information..........................................................................................................................155.1.4 Radar track......................................................................................................................................165.1.5 Label anti-overlapping....................................................................................................................175.1.6 Trajectories.....................................................................................................................................17

5.2 INSTRUCTIONS INPUT.............................................................................................................................185.2.1 Vehicles pie menu............................................................................................................................185.2.2 Route input......................................................................................................................................195.2.3 Clearances.......................................................................................................................................21

5.3 WARNINGS AND ALARMS.......................................................................................................................225.4 TAXIBOTS MANAGEMENT......................................................................................................................23

6 FURTHER IDEAS.......................................................................................................................................25

6.1 BIMANUAL INTERACTIONS.....................................................................................................................256.2 TANGIBILITY INDUCED PAPER STRIPS PROPERTIES.................................................................................256.3 COORDINATION WITH OTHER ATCOS.....................................................................................................256.4 HAPTIC FEEDBACK.................................................................................................................................256.5 PHYSICAL INTERACTORS........................................................................................................................256.6 LABEL ANTI OVERLAPPING....................................................................................................................256.7 FUTURE SITUATION ANALYSIS................................................................................................................256.8 MULTIVIEW............................................................................................................................................26

7 CONCLUSION.............................................................................................................................................27

8 REFERENCES.............................................................................................................................................28

3 of 30


acknowledged.


List of tables

Aucune entrée de table d'illustration n'a été trouvée.

List of figuresFigure 1: AVISO Radar image................................................................................................................8Figure 2: Paper strips............................................................................................................................ 9Figure 3: Strip board in Roissy CDG tower............................................................................................9Figure 4: Map of Roissy-CDG used in simulations..............................................................................10Figure 5: Matsuri radar image..............................................................................................................14Figure 6: Mini strips list........................................................................................................................15Figure 7: System information messages..............................................................................................16Figure 8: A/C icons Jumbo/Heavy/Medium/Light.................................................................................16Figure 9: taxibot and towed a/c icons..................................................................................................17Figure 10: Trajectory in edition.............................................................................................................17Figure 11: Validated trajectory.............................................................................................................18Figure 12: Marking menu (from [4]).....................................................................................................18Figure 13: Vehicles pie menu...............................................................................................................19Figure 14: Sticking line algorithm applied to route input......................................................................19Figure 15: Stylus.................................................................................................................................. 20Figure 16: Runway entry validation......................................................................................................20Figure 17: Parking entry validation......................................................................................................21Figure 18: Follow a/c clearances.........................................................................................................22Figure 19: Conflict display...................................................................................................................22Figure 20: Rule violation...................................................................................................................... 23

4 of 30


acknowledged.


1 Introduction

1.1 Purpose of the documentThis document describes the design process of Matsuri, the Ground Controller Working Positiondeveloped in project MoTa. Matsuri aims at achieving three main goals:

● Capture a maximum of information on the ATCo’s intentions with a minimum additional effortfrom him/her.

● Assist the ATCo in his/her tasks by providing relevant information or suggestions and byraising pertinent warnings. This second task relies highly on the quality of the informationinput, as the quality of the information provided to the user is dependent on the quality ofinformation provided to the algorithm.

● Assist the ATCo in managing a mixed fleet of a/c, taxibots, and a/c towed by taxibots. Sincetaxibots have yet to be actually deployed in airports, working hypothesis regarding theirfunctionality and behaviour were drawn from intermediate results of on-going studies1, ATCosinterviews and workshops.

The first chapter will describe the global technical choices made for the platform. Then the usagescenarios will describe the needs that should be fulfilled by detailing the main situations to deal with.Once the environment is set up and before digging further in the features, the second chapter givesthe design guidelines that were applied during the development of the interface. These guidelinesemerged all along the project but are described beforehand because they are relevant in the designchoices made to fulfil the features.From the usage scenarios were extracted a set of functionalities, supplemented by several ideascollected during design workshops, ATCos interviews on site or during the first simulation sessions.The third chapter presents these functionalities and explains how Matsuri helps the ATCos accomplishit. This chapter also details some functionalities that are not met or design ideas not pursued.Finally the last chapter will present the features ideas that emerged during the process and could beused to further enhance Matsuri interface.At some points of this document, a comparison to SESAR project 6.7.2 choices is made. SESARproject 6.7.2 aims at defining high level operational requirements which are implemented in otherprojects. The comparisons will be limited to the HMI implementations known to us.

1.2 Acronyms and Terminology

Term Definition

ATM Air Traffic Management

E-ATMS European Air Traffic Management System

SESAR Single European Sky ATM Research Programme

SJU SESAR Joint Undertaking (Agency of the European Commission)

SJU Work ProgrammeThe programme which addresses all activities of the SESAR JointUndertaking Agency.

SESAR ProgrammeThe programme which defines the Research and Development activitiesand Projects for the SJU.

1 Airbus currently work with ADP on impact evaluation and business model of taxibots through fast-time simulation.

5 of 30


acknowledged.


Term Definition

6 of 30


acknowledged.


2 Overall technical considerations

2.1 Hardware choicesThe Matsuri interface was designed on a Wacom Cintiq 27 QHD Touch stylus and touch screen.Wacom Cintiq devices became standard for ATCo working position due to their superior precision andhigh-end support. We opt for a Cintiq 27 QHD Touch which provides a large surface, but each screenarea is still reachable by a single user sitting in front of the device. Another reason for choosing thissize is the goal to substitute both vertical ground radar screen and the flat stripboard working surface.We selected the touch-capable device to allow multi touch and bimanual interactions, the stylus beingused for precise inputs such as route drawing.

2.2 djnn interactions libFor the development of the interface we chose the djnn development framework, a research toolkitcurrently being developed at ENAC. In addition to the support we can get on it, djnn is particularlyadapted for the project since it allows concurrent development of the interface with a graphic designerworking in parallel and it supports different interactors that could be used on MoTa. Indeed djnnframework makes an extensive use of SVG graphics, which allows to upgrade graphics design inparallel to the development (assuming that some naming rules are followed).

7 of 30


acknowledged.


3 ContextIn the framework of project MoTa, we chose to work on an airport sufficiently important to need a taximanaging system. For practical reasons of access to data, information, and ATCos experienced onthe platform, Roissy-CDG airport was selected.The MoTa project is focused on the Ground controller position, although there are different designopportunities with respect to collaboration with other sectors, grouped positions, sequencingmanagement. At Roissy CDG airport, there is a specific control position, the Apron controller, incharge of the a/c in the parking zones. This position could benefit from a Matsuri-like interface tomanage simultaneous pushbacks, use of parallel lines on the taxiways, and coordination with ground,but this is outside the scope of the project.Based on observations of ATCos in working situations at Toulouse and Roissy-CDG airports andinterviews with ATCos, we built up different usage scenarios marking out the use cases to bedesigned. These scenarios were used as a starting point for design workshops.

3.1 Working environmentThe starting point of the design process was the technology currently in use in control towers:

● Ground radar image (not interactive):○ Visual differentiation of arrival and departures a/c○ Automatic/manual labelling○ AVISO radar image in use at Roissy CDG:

8 of 30


acknowledged.


Figure 1: AVISO Radar image

● Paper strips with flight information. Strips give a/c identification information and specificinformation for arrivals (exit taxiway, parking entry point…) and departure (runway, holdingpoints…)

9 of 30


acknowledged.


Figure 2: Paper strips

● Strip board: geographical organization of paper strips according to flight characteristics andprogress.

Figure 3: Strip board in Roissy CDG tower

● Flight Plans are handled on a dedicated tool (DISCUS) for DEL ATCo: PLN activation, PLNawake/activated, departure clearance information, etc. It also gives details on the rolling time(block wait + rolling + rwy wait) and gives startup time

● Runway incursion safety net (does not concern GND position though)● General information always available: wind, QNH, current ATIS, current time, runways in use

(SIGMA interface)

We also noticed a few habits that would be interesting to retain in a digital environment:● paper strips for departure are physically given from GND to LOC ATCo● GND strips are used mainly to trace the holding point given to the pilot, to get the a/c callsign,

get parking information for arrivals, to keep record of the frequency statuses of a/c.● during traffic peaks, an assistant helps GND ATCo with sorting the strips on the board.● 80% of routes given to pilots are standard routes, i.e. routes that follow the traffic flows

defined on the platform.The other 20% are modified routes that, for instance, could requiretaking a taxiway against the normal flow of traffic.

● GND ATCo dispatches the departure a/c on the holding point according to weight categories,SID…

10 of 30


acknowledged.


The clearances used are listed below. The goal of the Matsuri interface is to give the possibility to theATCo to provide the same information to the electronic system:

● Pushback approved facing South/East (used in parking area)● Taxi via taxiways XX, YY, ZZ● Maintain holding-point XX● Follow a/c XX● After a/c XX taxi via...● Give way to a/c XX● Contact TWR 118.95● Stop immediately

Figure 4: Map of Roissy-CDG used in simulations

3.2 Arrival scenarioActor: Anna, ATCo at Roissy Airport.Location: Roissy South Tower, GND position.Context: parallel runway operation, average traffic, one GND position, fair weather, WESTconfiguration.

Anna sorts arrival strips by alphabetical order on the left of her stripboard. Strips on the stripboardmirror the planes currently on frequency.

Anna monitors the AVISO radar screen, she spots two arrival planes waiting to cross inner runway26R: BAW307 on S3 and DAL27 on S2. She extracts the two corresponding strips from the arrivalstrips stack and lookup their destination: Speedbird is going to stand A04, Delta to E26.The two paths cross each other, and there is a departure, AF1212, on F that might interfere on R.[She resumes visual scanning]

Local controller announces he sends the two arrivals. Anna throws a glance through the window toestimate the position of the Air France and respective aircraft speeds. BAW307 calls on frequency,she takes the strip in her hand and gives the clearance “Taxi straight ahead to stop RP1, contactApron on 121.925”. The pilot reads back, she drops the strip in the strips trash. DAL27 calls for taxi.She estimates that this one could not pass before the Air France. The Speedbird only crosses Rwhereas DAL27 has to use it. She lays the strip on the “R area” of the stripboard. She gives theclearance “DAL27, after Air France A320 coming from your left, taxi R to stop RP3”. Pilot reads back.She then informs the other pilot “AF1212 number one before Delta B767 on your right”.

[She resumes visual scanning]

11 of 30


acknowledged.


When DAL27 is close to RP3, she transfers it to apron. “DAL 27, contact Apron on 121.925”. Once thepilot reads back, she drops the strip in the strip trash.

3.3 Departure scenarioActor: Daniel, ATCo at Roissy AirportLocation: Roissy South Tower, GND position.Context: parallel runways operation, average traffic, one GND position, fair weather, WESTconfiguration.

Daniel sorts departure strips above his stripboard, mapping the actual position of parking area.

AF676SD request for taxi, he searches the strip and places it on the stripboard. He looks at therunway and gives the clearance “AF676SD taxi E, E5 for holding point W10 runway 26R”, he circlesthe holding point on the strip.

[Later]AF676SD calls back “AF676SD, we arrived at W10”, Daniel crosschecks the position of the aircraftand transfers it to the local controller, “AF676SD, maintain holding point W10 and contact Tower120.900”. Once the pilot reads back, Daniel dump the strip.

3.4 Incident scenarioActor: Inès, ATCo at Roissy AirportLocation: Roissy South Tower, GND position.Context: parallel runways operation, average traffic, one GND position, fair weather, WESTconfiguration.

Inès notices a plane stopped on E5, 20 seconds later, the plane still has not moved. She calls: – AF599F, do you have a problem? – we have a mechanical problem on the nose wheel, AF599F – do you need a tractor? – not yet, we are rolling checklists. – do you have any idea of the required time? – we will know in a minute how long we need. – OK, please call back then.

Inès calls a plane arriving behind the stopped planed “KLM12P, due to a stopped airplane, sidestepon the right as soon as possible to join E4, remaining path unchanged”.

AF758KG request taxi from P4 to 26R, she gives a clearance avoiding the stopped aircraft andemphasize instruction toward the taxiway used in reverse way “taxi second left on E4, then R forholding point W9, runway 26R”. She double-checks that no planes will enter E4 on the oppositedirection.

AF599F calls and announces that it might be able to restart on its own in ten minutes. Inès preparesfor a long unavailability of the taxiway and provides clearances using E4 to all new airplanesrequesting taxi.

3.5 Taxibots scenario

Taxibot operations are still being studied by manufacturers, airport managers, and airlines. For theneed of this project we assume the following operation frame:

taxibots fleet management (which aircraft uses taxibot, destination picking of empty taxibots)is out of ATC scope

taxibots are used only for departure

12 of 30


acknowledged.


taxibots detach from a/c on deicing pads

empty taxibots use the taxiways network even while not towing aircraft (worst case for ATCbecause of the extra traffic, the other solution being using service roads).

BUT empty taxibots dispose of service roads in the immediate vicinity of deicing pads to jointhe taxiways network via the closest parking area, parking J, instead of merging against thetide in the hotspot near holding points.

taxibots are unmanned

Actor: Thierry, ATCo at Roissy Aiport.

Location: Roissy South Tower, GND position.

Context: level 4 A-SMGCS, parallel runways operation, average traffic including departure towed

by taxibots, one GND position, fair weather, WEST configuration.

On the radar screen, AF788UM starts to blink on GE5, an icon figures that this plane is towed by

taxibot. Thierry points the aircraft and is proposed a default route that mainly respects one-way

taxiways except two sidesteps to avoid a roadwork area. This difference with default path is

emphasised. Thierry accepts the proposed trajectory, which is sent to the pilot of the plane. The

trajectory ends on a deicing area, near the 26R threshold. 7 seconds later, on pilot WILCO, the

trajectory turns green and disappears.

A message pops in the notification area, Thierry highlights it, KLM12P, which is towed by a taxibot on

R, requests a sidestep to the right on T, and expects to gain 50sec with this new trajectory. Thierry

understands that this manoeuvre allows overtaking an A380. He validates the suggestion (no wilco is

required).

A few seconds later a taxibot information message pops in the notification area, Thierry finishes giving

a vocal clearance and points the message. Taxibot 3 is coming back from 26R threshold (taxibot entry

point near J) to A stands. The expected trajectory is to use T-RT3-RP1 to join A stands. The

interacting flights (KLM12P going reverse on T and EZY 238X that might cross trajectory at RT3 / R

crossing) are highlighted. Thierry closes the notification, confident that the taxibot will avoid traffic.

He still monitors KLM12P going reverse and sees the taxibot holds before T to give way to the plane.

13 of 30


acknowledged.


4 Design methodologyMatsuri tries to be the missing link between the human operator and the electronic system. Wefocused on an ecological design to find interactions that would match the usual ATCo methodology.The first decision was to work on an interactive radar image, since this the central tool the groundATCo uses.From there on, we adopted a very exploratory approach to build up the features set. The final goalbeing the validation of route input methods and human/bots collaboration, we focused on the requiredfeatures to obtain an environment realistic enough to perform simulation sessions for the validation ofour proof of concept.The designed tool does not fully meet an A-SMGCS level [1]. However, Matsuri provides some level-4A-SMGCS features (route planning which can be downlinked to vehicles, conflict resolution service)but does not implement level-2 function runway incursion detection, as project MoTa focuses on theGND control position and not the LOC position.

14 of 30


acknowledged.


5 Features designIn this chapter are listed the different ideas that came up during the workshops and interviews inaddition to the standard actions that are required for the normal usage of the tool. For each one wegive an explanation on the retained design.

5.1 Information visualisation

5.1.1 Interactive radar imageDuring the first workshop, the idea came up of building a head up display for the ground controller toaugment the out of the window view with information specific to the a/c (color coding for departure orarrival) or taxiways and runways (superimposition of routes in case of low visibility). This would aproject on its own with the constraint of controller localisation in the tower and this was too complexfor project MoTa. Incidentally a study has been performed at CENA in 2006 detailing these difficultiesin project Survision[2].In the same workshop the idea of an isometric 3D view of the airport on the radar image waspresented. Even though the idea should be explored in greater detail, it has not been retained forMatsuri because of the problem it raises for pointing and designating objects of interests, the relativelylow position precision that could give a false impression when looking outside and trying to find thepresented 3D situation, and the lack of added value in a task that is predominantly 2D (the thirddimension coming in for the LOC position to visualize take offs and landings).Finally for the Matsuri interface we opted for a 2D view of the airport map as a background for theradar image.

Figure 5: Matsuri radar image

Screen layout:● the background is a map of the airport (here Roissy CDG south end) displaying the radar

track positions labelled with relevant information.● the upper banner (dragged from a drawer) giving access to general settings of the view, such

as the zoom level, the different layers of information to be displayed (parking zones name,adjacent sectors frequencies, buildings, taxiways names, etc)

● the right hand side banner is a list of flights (sorted in alphabetical order) which allows theATCo to quickly find the aircraft calling. When a flight is selected in the list, its track isparticularised on the radar image. This list also gives information about the flight.

● the left hand side panel holds different types of the messages : warnings of possible hazards(conflicting a/c, a/c deviating from clearance) and information on algorithm behaviour and

15 of 30


acknowledged.


decisions (autonomous vehicle trajectory update, potential vehicles separation issue or faceto face with a proposition to solve it, standard flow rules violations)

Interactions with a/c or taxibots are initiated by clicking directly on the vehicle icon, which pulls up apie menu, contextualised to the vehicle and its status, see 5.2. Instructions input.

As discussed in 2.1. Hardware choices, the tactile screen used for the controller seems wide enoughto display on a sufficient zoom level all the areas of interest for Roissy-CDG airport. This will beverified during the experimentation using this version of the tool. Anyway this may not be applicabledepending on the airport topography. As Roissy CDG is quite rectangular, it fits well on a wide screenwhereas other airports may not. We discuss this in 6.7. Multiview.

SESAR Project 6.7.2 implementations are using a two screens setup, one for the radar image and thesecond for electronic stripping. Only one known implementation uses a tactile interface for theelectronic stripping, all the others define the mouse as input system.

5.1.2 Paper strips replacementThe classical way to replace paper strips in an electronic environment is either electronic stripping, i.e.a list of strips equivalent to their paper version, or a stripless environment where the information andinteractions are implemented on the vehicles’ label. In Matsuri, it is a mix of both solutions. Thestripless version assists in maintaining a geographical organization of the strips on the board. But wealso had some remarks from controllers after the first simulations, who found it difficult to localize thestation calling on the map. Hence, a list of mini-strips has been added in a retractable side panel.When the controller hovers over a mini strip, the corresponding track is highlighted on the radarimage.

Figure 6: Mini strips list

Currently these mini strips provide more precise destination information once the controller has givena clearance. For instance, a departure a/c will have its departure runway replaced by the runway entrypoint once it has been specified by the ATCo. The mini strip will also contain the last order given to thepilot, more precisely the last clearance entered in the system for that flight.Then we can imagine a later evolution to sort these mini strips by holding points (instead of callsignalphabetical order), it will give the ATCo a better view of the forthcoming situation.

5.1.3 System informationThe left hand side pane described in the general layout is still a work in progress. It is the center ofcommunication between the ATCO and the system. Indeed the algorithm will give useful informationto the controller to help him understand its processing. The messages are stored in a retractablepanel that can be extended by the user when needed. In order to ensure that the ATCO does not miss

16 of 30


acknowledged.


an important information, the message items are slightly overstepping the panel so that the coloredbar coding their importance is always visible on the screen (see left screenshot below). When thecontroller swipes the panel to the right, the full messages appear (see right screenshot below).

Figure 7: System information messages

An incoming evolution is to give the ATCO the possibility to classify the messages depending on theimportance level he wants to give an item. Indeed the items can be dragged on different levels fromleft to right or completely deleted. A second evolution is for the potential conflicts messages to be keptin a minimized version until the conflict is actually resolved, thus the controller can keep in mind tomonitor it, even if a deviation warning will be raised if a vehicle does not follow the instruction to solvethe conflict.

5.1.4 Radar trackConcerning the a/c radar track icon, Matsuri proposes 2 design specificities:

- Icon sizing depending on wake turbulence category: it allows the controller to get idea of theaircraft size and gives a hint on the kind of a/c to look for when looking outside.

Figure 8: A/C icons Jumbo/Heavy/Medium/Light

- Analysing the variation of radar tracks position we can differentiate still a/c from moving a/cand apply a different design on them, thus giving a quick hint to the controller on the movingand still vehicles.

The taxibot icon is not to scale or else it would be too small and difficult to find on the radar image.

17 of 30


acknowledged.


Figure 9: taxibot and towed a/c icons

Furthermore the a/c icon color follows the codification below:

● : the a/c is under another controller’s responsibility

● : a previous controller has transferred the a/c to the ground frequency, the ATCO shouldexpect an imminent call from the pilot.

● : the a/c is currently under ground responsibility. It turns pink when the ATCO selects iton the interface and light pink during the route input process.

● : the a/c has been transferred. It does not go back to light gray until the next controllerassumes it as the ground controller should still monitor it.

5.1.5 Label anti-overlappingAfter the first simulations using the initial version of Matsuri, it was quickly determined that the radarimage could be cluttered with labels, especially during periods of heavy traffic. The first attempt atsolving this problem has been to reduce labels and expand them only on demand. In addition, thenew version will be testing an anti overlap algorithm based on the same concept as the sticking linedescribed in 5.2.2. Route input. This algorithm implements an anti overlap system between labels butalso with the a/c icons to avoid masking a potentially conflicting a/c when tracing a route, alsobecause some interactions on an a/c require designation of another a/c.In 6.6. Label anti overlapping are given additional anti overlap ideas that will not be tested in Matsuridue to the lack of time to implement them.

5.1.6 TrajectoriesThe trajectories entered by the ATCO can be entered via one of two representations:

● During the route tracing process, the trajectory is represented as a trace drawn by thecontroller

● As soon as the drawn trajectory matches a portion of the taxiway, feedback is given to thecontroller by replacing the hand drawn shape by the map shape of the taxiway.

The latter representation is also used once the route drawing process is completed to show ondemand an a/c route. Then a color code is used to differentiate the planned route, the cleared portionof the route, the route sent to the a/c by datalink.

A route being edited is yellow:

Figure 10: Trajectory in edition

18 of 30


acknowledged.


Once this route is validated, it turns green:

Figure 11: Validated trajectory

If the a/c is equipped with datalink then the trajectory is orange until the WILCO is received and turnglowing green.

5.2 Instructions input

5.2.1 Vehicles pie menuBelow are the different ideas that were proposed to the users and implemented for the vehicle’s piemenu:

● Pie "gate menu": a circular menu where each slice defines a direction to swipe and activatean action. It gives the possibility to finish the gesture by designating a place or another vehicleto specify the action (for instance give way to a given a/c).

● Marking menu: an experienced user can quickly access a function by making a straight markin the direction of the desired menu item without popping-up the menu.

Figure 12: Marking menu (from [4])● The pie menus should be consistent on every item, and menu items that are not meaningful

for an object should be greyed out.● Pie menu orientation could either be:

■ absolute in the screen reference■ follow a/c heading

The orientation following a/c heading has been abandoned because it makes it would forcethe controller to check the menu orientation before selecting an action and cannot benefitfrom learning the items position.

● Actions could be contextualised depending on the designated vehicle part:○ vehicle nose: define trajectory○ vehicle body : information○ vehicle tail: pushback / brake / stop

This requires an interaction very precise and it is compatible with the controller work flow.We kept the marking menu and gate menu:

19 of 30


acknowledged.


Figure 13: Vehicles pie menu

The “Frequency” menu item is used to assume or transfer an a/c, depending on its previous status. Itcan also been activated by a left swipe.The “Automatic” menu item requests the standard route (see 5.1.2. Route input). It can also beenactivated by a right swipe.The “Stop” menu item can be used for an immediate stop, or in the sub menu the request can bemore precise. “There” gives the possibility to enter a partial route clearance, “Follow” gives the orderto take the same route as a given a/c (see 5.1.3.2. Conditional clearances) and “Yield” specify to giveway to an a/c. Both Follow and Yield actions require the ATCo to designate a second aircraft. The taxibot menu is slightly different since it has additional taxibot specific actions such “Attach” toretrieve a specific a/c and tow it or “Resume” to restart its activities after an ATCo intervention. As longas the taxibots are not transferred to other sectors, the “Frequency” item is not accessible.

5.2.2 Route inputThe route input is the key element of the interface, as it is obviously the most frequent task to beaccomplished. The basic idea that initiated the project was an algorithm developed at ENAC, StickingLine, based on a physical analogy with hills and valleys. Taxiways can be considered as valleys andthe empty spaces between taxiways are hills. When the user ''drops" a gesture on this terrain, theinput falls into the valleys. This algorithm is a simplified version of the active contour used in imageprocessing [3]. One of the virtues of this algorithm is its transparency to users. Users can quicklyunderstand the cause of algorithm misinterpretation of the users’ input. To supplement the user'scomprehension, we chose to animate the algorithm convergence process.

Figure 14: Sticking line algorithm applied to route input

On the above screenshot (Figure 14), the orange line is the result of a quick drawing on the interfacewhich is progressively sticking, or adhering, itself to the taxiways. This process allows the ATCo to directly draw on the map the directions given to the pilot: a quickgesture will not be precise and it will take the closest taxiways, a slower gesture gives the possibilityto make precise turns on the taxiways.Despite the capability to trace a route on the map for a given a/c, ATCos had a lukewarm reception.Afterwards it is quite obvious that it would be tedious to draw the complete trajectory for all a/c, since80% of the time at least a large portion of the route is a standard route.

20 of 30


acknowledged.


Based on this feedback, we decided to use the software designed for the MAS part of project MoTa,which is capable of computing the standard route for a given a/c with few information(arrival/departure, a/c type, destination, airport configuration). Therefore the outcome is of a mixedinteraction where the ATCo can start drawing a path, for instance to extricate an aircraft out of acomplicated situation, and at any time ask Matsuri to complete the trajectory with the standard routefrom the end of the hand-drawn route to the destination via the pie menu.This new version was better appreciated by ATCos. One of them told us that he would feel morecomfortable at tracing the a/c route by clicking on particular point of the trajectory instead of drawingit. This interaction would be more adapted to the usage considering that the route is given to the pilotover the radio. Indeed, the radio message would be “Taxi via Fox and Romeo to Romeo Papa 6”. Byclicking respectively on taxiway Fox, taxiway Romeo and the Romeo Papa 6 holding point, it shouldbe possible to compute the standard route passing through these points.Hence, we mixed these three solutions so that the ATCo can trace a precise route using the stickingline or simply designate points of the trajectories or directly request the standard route. Indeed, thesethree modes can be mixed in the processing of a single trajectory:

● start with a drawing to precisely circle around a blocked taxiway,● add a few points to get back on a standard taxiway,● request the standard route to complete the trajectory.

In addition, we relied on the parallel between the tablet stylus and an actual pen to enter/exit the routedefinition process or erase the route (using the “eraser” part of the stylus). Although the tablet can beused as a touch input device, the route tracing is only available with the stylus because it is a fairlyprecise input. We use the button on the stylus to enter/exit the route edition mode.

Figure 15: Stylus

The downside of the stylus button is that it requires using the stylus in a precise orientation, otherwisethe stylus button could be not reachable, or pressed inadvertently. We are thus also considering alogical button on the interface to switch modes (edit/display) that would always be available. At first,we tried to avoid bi-manual interaction for the path input since usually the ATCo will talk on the radiosimultaneously, thus having one hand used for the push-to-talk and the other with the pen. This bimanual interaction would be limited by a push to talk system in the hand. Finally we had to work on an interaction to validate the route. Even automatic standard routes needvalidation to designate the final holding point.

Figure 16: Runway entry validation

As shown on the above screenshot, a departure route in the west configuration can lead to threedifferent holding points on runway 26R entry. Validating the route is easily done by clicking on thedesired waypoint. In addition, the ATCo can be presented an indication on the holding point load. Themulti agent system monitor the expected load on each holding point and provide the predicted load at

21 of 30


acknowledged.


the time the a/c being handled will reach runway threshold. This information is represented to theATCo by colour coding the estimated foreseen number of waiting a/c, giving him hints for choosing anentry point.Arrival a/c have the same validation needs, but instead of runway entry point the parking entry pointare used to validate the route. Although the system knows to which parking area the a/c is heading, allthe parking entries are proposed to the ATCo in case of a change in the parking allocation orcoordinated use with apron controller of internal taxiways.

Figure 17: Parking entry validationSESAR project 6.7.2 HMI implementations usually propose point to point selection of the route on thetaxiway intersections. MoTa system is more flexible as it allows to click anywhere on the runways. Thedirect drawing and the mix of different inputs method as it is on Matsuri is not proposed on 6.7.2 HMI.Moreover Matsuri allows route modification by drag and dropping pinpoints that are placed along thetrajectory.

5.2.3 ClearancesThe primary objective we want to achieve in this interface is to give ATCOs the possibility to input asmuch information as they give to the pilots on radio. Hence the electronic system will have a sufficientlevel of knowledge on what is happening on the platform to provide the ATCos with relevant routessuggestions (based on the current traffic load, possible incidents) and pertinent warnings on potentialhazards. However, this information must be inputted into the screen as the electronic system does nothave access to what is communicated on the radio.

5.2.3.1 Datalink vs. radioSince we are considering a futuristic system, we worked on the question of the datalink usage onground. Whereas SESAR project 6.7.2 considers Taxi-CPDLC for initial clearance (at parking) andvoice communication then, we extend the use of datalink to any taxiing clearance (including priority forcrossing or route modification) for equipped aircraft. This choice is first motivated by consistency sincetaxibot will be datalink-only it seems coherent to allow the same scope of clearance for datalink-capable aircraft; and second by reasonable expectations that shorter communication delay will beavailable (at least on airport surface) at the studied time.In Matsuri we want the ATCo to specify every clearance given on the radio in the electronic system. Inaddition to feeding the algorithm, this will allow a better use of datalink. We believe the ATCo will beable to handle more aircraft than the radio system.An important consideration was how to differentiate urgent messages from others. Therefore the“Stop” menu item does not send a datalink request; instead the ATCo must talk directly to the pilot toget a quicker response. The menu item is present and should be used only to inform the electronicsystem that the a/c has been requested to stop so that it can be taken into account in the algorithmswhen needed, for instance to avoid warnings on route deviation.

5.2.3.2 Compound clearancesIn the perspective of an ecological design, we feel that it is important to give the ability to build morecomplex clearances that are commonly used by the ATCo:

○ X follow Y ○ X give way to Y / X after Y taxi via ZZZ to VV○ Stop There

Compound clearance means a clearance that needs detailed information such as a second a/c or ageographical point.

22 of 30


acknowledged.


The “follow” and “give way” clearances are given by selecting “Stop->Follow” or “Stop->Yield” in thea/c pie menu and then point the a/c to follow or the a/c to give way to.

Figure 18: Follow a/c clearances

On this screenshot, NWA874 has received the clearance to follow BEE493K.Even though equivalent clearances can be set using existing interactions, these are “shortcuts” andfollow the ATCos habits. Like all other clearances, once entered in the electronic system they give thealgorithms a better background on the on-going situation and they may be able to raise relevant andspecific warning such as an a/c that was supposed to give way to another does not stop at thecrossing.SESAR project 6.7.2 does not propose these compound clearances but sticks to the officialphraseology. The idea of such clearances came up in MoTa project while observing the ATCos duringthe first simulations batch (using current working techniques), these clearances save time on theradio.

5.3 Warnings and alarmsMatsuri notifies the controller of different types of warnings and alarms:

● conflicting validated route detected● deviation of clearance● vehicle breakdown detected● route violation of standard airport process.

The main concern on this topic was the number of warnings that could be raised, especiallyconcerning conflicting routes. The approach to refine the pertinent conflicts will be discussed indocument D4.2 - AMAS Opt ATC Use case adaptation Final. On the HMI side we chose to split theconflicts display in two:

● immediately when it is detected, an icon on the intersection is displayed and the a/c arehighlighted. This warning can be quickly acknowledged by clicking on it, it disappears to avoidoverloading the radar image.

● in the side panel, the conflict is kept in the list and is shown on the radar image by hovering it.

23 of 30


acknowledged.


Figure 19: Conflict display

In the warning message, a proposed solution is displayed. The algorithm proposes one a/c to givepriority and a reason to justify the choice. The different reasons have been studied in a side studyduring which around 30 ATCOs have answered an online survey where they were presented withdifferent intersection situations and ask for their choice of priority. Nevertheless if the priority given bythe algorithm does not suit, the controller can change it by selecting the a/c in question in the wishedorder of priority. Datalink a/c would then be notified with the decision while others are warned by theATCO over the radio. Regarding route violation of standard airport process, the algorithm provides the rule which is brokenwhen an ATCO draw a route that does not follows the standard flows defined on the airport. It isalways possible to input such a route, validate it and send it through datalink because the ATCO hasthe final decision, on the HMI we simply highlight the portion of the route that causes a problem andput a message in the sidebar list with an explanation of the warning:

Figure 20: Rule violation

5.4 Taxibots management

Due to the worst case condition, with empty taxibot on taxiways, that might double the traffic for everytowed plane, we opted in consultation with ATCos for a high level of autonomy for taxibots: they picktheir path by themselves, notify ATCos, but do not require validation of ATCos before they move. In away, they behave like sweeper truck that ride among the traffic and avoid aircraft by themselves, incontrast they are clearly visible on ground radar screen and ATCos can know their goal and plannedpath.● when a taxibot tow an aircraft, the aircraft is controlled as any datalink-capable aircraft (that isclearance provided preferentially by datalink, emergency clearance -STOP- by radio telephony)● when a taxibot is empty, the default behavior is to find its path on its own.

○ taxibot can pick another path at any moment,

24 of 30


acknowledged.


○ they are in charge of avoiding others vehicles○ ATCos can watch the current intention of taxibot at any moment.○ ATCos can still give clearance to a taxibot as to any datalink aircraft, in this case, thetaxibot does not change its path anymore.

On Matsuri tools, taxibots appears as aircraft with an icon and a label. The label displays the taxibotname as well as its objective if he has one (the callsign of a plane to pick or a geographical target).When a taxibot decides to change its path, the new path appear with a low-priority in the notificationarea.

Towed A/C offer the same interaction set of aircraft, plus a detach action to allow to detach at anyplace (eventually outside the deicing pad). This feature might allow ATCo to perform micro-optimisation by briefly blocking a taxiway for detaching if there is no traffic behind.

Taxibots allow two more order than regular aircrafts: resume, which is meant to be used after a stop toallow the taxibot to restart taxiing in autonomous mode (remember that once given a taxi clearancethe taxibot strictly follow this clearance) and attach to send the taxibot pick an aircraft (note: this lastfeature target fleet management and is not required for ATC, it was added in a prospective way).

25 of 30


acknowledged.


6 Further ideas

6.1 Bimanual interactionsDue to lack of time, bimanual interactions have not been further analysed. They are only used as aswitch between the normal mode and the route edition mode to replace the stylus button.bimanual interactions could be interesting in case of giving orders to a group of vehicles, one hand toselect the vehicles and the other to define the order.

6.2 Tangibility induced paper strips propertiesSwitching to an electronic environment presumably induces the loss of interesting properties oftangible artefacts such as paper strips. We did not further study this point but we have identified atleast two occasions when it could be useful:

● ATCO shift change● Transfer from ground control to LOC or apron

6.3 Coordination with other ATCosSince we focused on the ground controller, we did not study the interactions with other controllers, butthere are some:

● Ground and apron negotiation on parking entry / exit that do not follow the airport’s standards● Ground control preparation of departing sequence for LOC and collaboration over a DMAN● Taxiing impacts of an arrival sequence change, on parking availability for instance

6.4 Haptic feedback Haptic feedback could be interesting on different occasions:

● stylus vibration when tracing a conflicting route● surface modification to guide a selection of a runway entry point (surfpad, Roussel &al,

inflatable buttons, Harrison &al).

6.5 Physical interactorsPhysical interactors can find their place on the interface:

● to block a resource, as some LOC ATCOs put their arm on the runway during a takeoff orlanding to avoid authorizing a crossing

● to remind the ATCO of on-going maintenance operation on a sector.● degraded mode

6.6 Label anti overlappingAnti-overlapping process could be completed with specific features during the route tracing processsince the taxiways should be free to facilitate the drawing but the ATCO still needs to see the potentialobstacles or conflicts. In addition the cursor could spread the offending labels away from the taxiwayor the opacity of other items could be adjusted during this process.

6.7 Future situation analysisA wide field of investigation we did not have time to delve into is the representation of probable futuresituations. Several possible interactions have been cited during workshops:

● double designation: where will be other a/c when this one will be there ? (when the ATCOselect an a/c and one predicted future position)

● dragging the a/c along its predicted trajectory to see other vehicles moving on their ownprediction.

● timeline to travel in time through a slider or jog wheel. Either all the a/c are represented or theATCO selects few interesting a/c for a specific situation analysis.

26 of 30


acknowledged.


● present and future situations need to be compared so the present situation representationmust be adapted when looking at future situation: different colour, ghost effect, blur, opacity...

Design ideas taken from ERASMUS[5] or MAMMI[6] to represent incoming events in an agenda andtask/clearance scheduling on a time moving “conveyor belt” were also considered for integration intothe interface. This would give the controller a view on incoming potential conflicts and prepareDatalink clearances to be sent later.In order to simplify each view, it could be interesting to build up different timelines for different needs:

● Departure / arrival sequences● Vehicles passing at a given intersections (hotspots before runway entry points,

holding points to CDG north ground sector…)

6.8 MultiviewSome ATCOs expressed the potential need of different views of the situation, such as having differentzoom levels depending on the zone, a fish-eye/wide angle view of zone of interest or a synthetic viewof the airport where zones are contracted depending on their interest (for instance, long straighttaxiways could be shrinked).We did not work on that particularly because the size of the screen allows us to display the wholearea with a sufficient zoom level.An interesting idea to be further studied for the interactions between controllers is to have a mobilecopy of the interface that would allow some to see the situation without disturbing the controller incharge. It could be useful to prepare a shift change for instance.

27 of 30


acknowledged.


7 ConclusionThe HMI resulting from this design process will be validated during 2 sessions of simulations inOctober and November 2015. The first of these two session focus on the tool itself, the interactionsand the algorithms, to validate that it is actually usable and does not affect the ATCO workload andhope fully assist him correctly in his tasks. This experiment does not include taxibots. The taxibots willbe introduced in the second simulation. The benefits on the global taxiing time can then be analysedseparately.

The different interactions and features of the HMI have been guided by a number of discussion withATCOs but also by the emerging possibilities offered by the algorithms developed in parallel. Thework has been highly iterative and choices had to be made to focus on the most useful features.

The feedbacks gathered after these sessions will help confirm the design and provide insights for thefinal prototype. This document will have to be updated afterwards.

28 of 30


acknowledged.


8 References[1] Eurocontrol Definition of A-SMGCS Implementation Levels, 1.2, 30/06/2010 -

https://www.eurocontrol.int/sites/default/files/field_tabs/content/documents/nm/airports/a-smgcs-definition-of-implemetation-levels-v1-2-20100630.pdf

[2] DTI/SDER/N.Desmuyck Mémoire de Qualification Technique Supérieure, Application despossibilités de la réalité augmentée au contrôle aérien en vigie : Illustration par utilisation d’undispositif de visée tête haute, NT06-239, Aout 2006 - http://amisducena.fr/output/NT06-239.pdf

[3] Kass, M., Witkin, A., and Terzopoulos, D. Snakes: Active contour models. InternationalJournal of Computer Vision 1, 4 (1988), 321–331.

[4] Bill Buxton User Learning and Performance with Marking Menus -http://www.billbuxton.com/MMUserLearn.html

[5] ERASMUS project : http://www.atm-erasmus.com/

[6] MAMMI project: http://lii-enac.fr/en/projects/mammi/

29 of 30


acknowledged.

http://www.atm-erasmus.com/

http://www.billbuxton.com/MMUserLearn.html

http://amisducena.fr/output/NT06-239.pdf

http://amisducena.fr/output/NT06-239.pdf




-END OF DOCUMENT-

30 of 30


acknowledged.

title of the documentihmaero.recherche.enac.fr/data/documents/e.02-24... · d2.2 – functional...

Documents