wp4 – r4 – demonstrator: implementation and source code · 2007-11-12 · wp4 – r4 –...

RESEARCH & TECHNOLOGY FRANCE

Source Version Date Page

INTUILAB, THALES R&T 1.1 2004-12-20 1/22

Reference TRT-Fr/DAS/HIT/OG,04/181 © European Organisation for the Safety of Air Navigation (EUROCONTROL) June 2004

EUROCONTROL

EUROCONTROL

SCOPE – CARE II Innovative WP4 – R4 – Demonstrator: Implementation and

source code

Thibaut EHRETTE & Alexandre LEMORT

This document is published by EUROCONTROL in the interests of the exchange of information. It may be copied in whole or in part, providing that this copyright notice and disclaimer are included. The information contained in this document may not be modified without prior written permission from EUROCONTROL.

EUROCONTROL makes no warranty, either implied or express, for the information contained in this document, neither does it assume any legal liability or responsibility for the accuracy, completeness or usefulness of this information.

WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative

Versions

N° Date Writing Review Correction

1.0 2004-11-26 T. EHRETTE (THALES R&T) & A. LEMORT (INTUILAB)

1.1 2004-11-30 T.EHRETTE T. EHRETTE

Circulation

Addressee Version Date

Marc BROCHARD (EUROCONTROL) 1.0 2004-11-26

Validation

Name Date Signing

Written by: T. EHRETTE, A. LEMORT 2004-11-26

Agreed by:




EUROCONTROL




Table of Contents

1. Introduction................................................................................................................................0

2. Recognition process ...................................................................................................................0

2.1. Grammars ............................................................................................................................0

2.2. Interpretation .......................................................................................................................0

2.3. ASR result logger ................................................................................................................0

3. Description of scenarios.............................................................................................................0

3.1. No dialog engaged...............................................................................................................0

3.2. Watermarking plugin only...................................................................................................0

3.3. Clearance/acknowledgement plugin....................................................................................0

3.4. Message inconsistencies feedback ......................................................................................0

4. Conclusion .................................................................................................................................0

5. Appendix A: Controller Grammar .............................................................................................0

6. Appendix B: Demonstration scenario ........................................................................................0




EUROCONTROL




1. Introduction This report (R4) concerns the last work-package (WP4) of the CARE II Innovative project SCOPE (Safety of COntroller-Pilot dialoguE). The work presented here has been done by THALES Research & Technology (THALES R&T) in collaboration with IntuiLab. This document presents the scenario of the demonstration and the development of the specific modules such as grammars or interfaces to make it feasible. The goal of the demonstrator is to present a possible use of the speech recognition during the controller-pilot dialogue and to show that this technology could improve its safety. To simplify the task and highlight the characteristics of the system, the dialog has been reduced to one or two orders given by the controller and acknowledged by the pilot; this represents the majority of the pilot-controller speech exchanges. A new grammar has been designed to achieve the previous one (described in WP3-R3) adding the whole phraseology to the only callsign, and an interface program (written en java) has been developed to run speech recognition with the appropriate grammar and send the right formatted interpretation to the other modules of the demonstrator via the Ivy bus. The recognition works on two PCs under Microsoft Windows 2000: in one side for the controller and in the other side for the pilot. SCOPE's demonstrator visual interface is composed of three components:

• A radar display upon which are plugged three plugins. Those plugins aim at assisting the controller during the controller-pilot dialog;

• An air-traffic control simulator communicating with the radar display; • An interface, called logger, presenting textually the results of the two ASRs (controller &

pilot) and logging them into a textual file which can be consulted after the demonstrations. The radar display and the A.T.C. information simulator are running on a Linux based computer. The logger is running on one of the two Microsoft Windows computers running the ASR The demonstrator displays feedbacks on speech recognition results to the controller during the dialog, permitting to highlight human understanding errors before they can cause warn.

2. Recognition process The first step of the demonstrator is to recognize the controller and the pilot clearance. This implies two microphones linked to a specific computer simulating the controller and pilot interface. Each computer has its own recognition system with a specific context-free grammar (CFG) as shown in figure 1.




EUROCONTROL




Figure 1: Modules of the recognition system

2.1. Grammars A grammar is inherent to any constraint vocabulary recognition system, as Nuance, and specifies the whole possible utterances that can be recognized and their interpretation (Nuance slots structure). For the demonstrator, a clearance is composed of a callsign followed by one or two orders. These orders concern the flight level, the cap or the speed represented by three sub-grammars called FL_ORDER (for Flight Level order), CAP_ORDER and SPD_ORDER (for Speed order). All grammars are written in GSL (Grammar Specification Language, see R3). FL_ORDER [ ([climb descend] to flight level Number_FL) ([climb descend] to Number_feet feet) ] CAP_ORDER ( turn [left right] heading Number_360 ) SPD_ORDER ( speed [(Number_mach mach) (Number_knots knots)] )




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative In order to improve the recognition results, constraints on numbers have been added because the value of cap and speed cannot be any integer. Number_FL, Number_feet, Number_360, Number_knots and Number_mach are five constraint grammars deriving from a more generic one, called Number that covers any integers. For the demonstrator, the constraints are as follows:

• A flight level without unit precision is between 70 and 600 with a 10-step (Number_FL). • A flight level in feet is between 100 and 9000 with a 500-step (Number_feet). • A cap value is given by three successive digits : hundreds, tens and units (Number_360). • Speed values in knots are contained between 100 and 390 (Number_knots)Speed values in

mach are contained between 0.70 and 0.90 (Number_mach) These constraints affect the sub-grammar design. The controller’s grammar can be found in Appendix A. As the controller gives orders and the pilot acknowledges, they use different utterances but with very slight differences: the pilot’s answer is just a reformulation of the controller’s order with verbs in the continuous form (+ ing). The pilot’s grammar doesn’t imply any particular development.

2.2. Interpretation Whatever the way it is pronounced, the sense of each crucial keyword or group of words is carried by a slot. For example, the three following utterances express the same order (climb, flight level 41):

“climb to flight level forty one”

“climb to forty one feet”

“good morning, climb to flight level forty one due traffic” A slot is a key-value couple that represents in text the spoken information and can be easily formatted into an Ivy compliant string. The keys represent the type of order: climb, descend, turn-left, turn-right, speed-knots and speed-mach, and the value is the targeted flight level, heading or speed in knots or mach. The slot structure output string is supplied by the Nuance server as a result of the recognized utterance. For example:

“Air France yankee oscar” ⇒ {<callsign-slot AF259YO>}

“Air France yankee oscar climb to flight level sixty”

⇒ {<callsign-slot AF259YO> <instruction3 climb> <value3 60>}

“Air France yankee oscar turn-left heading two four five and speed three hundred knots”

⇒ {<callsign-slot AF259YO> <instruction1 turn-left> <instruction2 speed-knots> <value1 245> <value2 300>}




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative The conversion into an Ivy compliant format is now easy: ⇒ controller callsign=|AF259YO| instruction1=|turn-left| value1=|245| instruction2=|speed-knots| value2=|300| responsetime=…

In case of different possible interpretations, Nuance server gives the N-best solutions with a confidence score up to the rejection threshold. In order to reduce the number of interpretations, this threshold is fixed to thirty on a 0 to 100 scale (0: no rejection) and the maximum number of solutions is fixed to five. The output string is completed if values differs as follows:

“yankee oscar descend to flight level ninety” can be misinterpreted if the value is imprecisely pronounced:

⇒ {<callsign-slot AF259YO> <instruction3 climb> <value3 90>}, best solution, confidence score=70;

⇒ {<callsign-slot AF259YO> <instruction3 climb> <value3 19>}, second solution, confidence score=59. The Ivy conversion holds the ambiguity: ⇒ controller callsign=|AF259YO| instruction1=|climb| value1=|90|19| responsetime=…

2.3. ASR result logger During the demonstration, an interface permits to view the speech recognition results returned by the two ASRs For each recognized sentence, the displayed information is:

• Ivy messages exchanged on the network between the two ASRs and the controller position; • Clearance information: callsign, orders (parameter/value pairs); • Analyse duration.

A summary of the analysis duration is available for both pilot and controller ASRs, minimum, average and maximum analyse duration are displayed on the information area at the bottom of the screen. The figure 2 presents the interface of the logger running on a linux platform. Note: This interface is realized using the java programming language, during the demonstration, it should be used on a computer running Microsoft windows OS, the MS windows java look & feel may causes slight difference with the figure 2 illustration.




EUROCONTROL




Figure 2: The logger interface This interface also logs the results of the recognition processes in a file permitting the analysis of the demonstration results. The name of this file is displayed in the centre of the bottom area.




EUROCONTROL




3. Description of scenarios The controller position is based upon a traditional radar display. Three plugins are added to this position in order to help the controller in his task with the ASR results:

• An ASR status plugin; • A watermarking check plugin; • A clearance/acknowledgement match.

3.1. No dialog engaged As already mentioned, the controller's position interface looks like a traditional radar interface. Figure 3 shows this position while no dialog is engaged between the controller and pilots in his controlled area. On the top left corner of the controller's display, an area displays the states of the two ASRs The low level of opacity expresses that no dialog is engaged.

Figure 3: Controller's position without any feedback




EUROCONTROL




3.2. Watermarking plugin only A first part of the help provided to the controller is a check on the correspondence between the called pilot's callsign and the responding pilot's one. If there is match, an "OK" feedback is rendered on the track on radar display. Otherwise, the two tracks involved in this incorrect dialog are highlighted. The controller initiates the dialog calling a pilot, by example saying “AFR023,Paris center, good morning”. The ASR status area takes a lighter colour and becomes less transparent, showing that the dialog is engaged and a feedback for representing a speech recognition success is rendered in the controller's part. Moreover, on the radar display, the AFR023 is highlighted by a green rectangle surrounding the track's label. The feedback corresponding to this step of the dialog is shown on figure 4.

Figure 4: The called track is highlighted Two cases are then possible for the watermarking plugin:

• The designated pilot acknowledges the hello message answering by example: “AFR023, good morning Paris centre”. In this case, the pilot's part of the ASR status area takes the “speech recognition success” colour and the corresponding track is highlighted by a change of the size of the callsign police in its label. After a few seconds, the feedbacks on the ASR status area and the involved track are hidden, the ASR systems results permits to estimate that no errors were made during this controller-pilot dialog. The correct callsigns match feedback is shown on figure 5.




EUROCONTROL




Figure 5: The good pilot acknowledges the clearance

• The pilot of another aircraft acknowledges the message. In this case, the ASR state has the same behaviour than in case of callsigns match since the error is neither a speech recognition failure nor an inconsistency. The feedbacks of these incorrect comprehensions between the controller and the pilots are rendered on the radar display by the watermarking match plugin. The clearance track vibrates indicating it was involved in the dialog and an animation is rendered on the track corresponding to the acknowledgement callsign: a red rectangle grows from the centre of its label until it completely surrounds this label. The red colour is used as an attention getter for the controller who must contact the two involved pilots in order to correct the understanding error detected by the speech recognition process. The figure 6 is an example of the callsigns mismatch. It illustrates the ends of the two animations on the involved tracks.

Figure 6: Another aircraft acknowledges the message

3.3. Clearance/acknowledgement plugin If the controller gives order(s) to a pilot in a clearance, the orders are rendered on the radar display. In SCOPE's demonstrator, a clearance is composed of a maximum of two orders. A summary of these orders is displayed just bellow the track's label by the clearance/acknowledgement match plugin. This feedback aims at helping the controller to verify the ASR results and to remember the




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative orders that are waited in the pilot's acknowledgement. The feedbacks for clearance with respectively one and two orders are shown by figures 7 and 8.

Figure 7: Clearance with one order

Figure 8: Clearance with two orders On reception of the pilot acknowledgement, if the callsigns match test is successfully passed, the acknowledgement is submitted to another test: parameters and values of orders must match the ones pronounced in the controller's clearance. If the clearance/acknowledgement test is passed successfully, the plugin displays a feedback on the corresponding track: a slight change is made on the summary already displayed under the track's label. This correct acknowledgement feedback is shown by figure 9.




EUROCONTROL




Figure 9: Clearance/acknowledgement match If one or both received values do not correspond to the waited ones, the clearance/acknowledgement match plugin displays a mismatch feedback on its already displayed summary: the summary's surrounding box becomes red and the received value is printed in red close to the waited one. This feedback permits to see at first sight the conflict between the clearance and the acknowledgement and should help the controller to correct this potentially unsafe situation.

Figure 10: One value of the acknowledgement mismatch the clearance If it is the parameter of one or both orders that causes the mismatch, the error feedback hides the value of the involved parameter(s). Indeed, in this case, a possible value error can be put aside. The feedback highlights the difference between the waited parameter and the received one. This feedback is shown by figure 11.




EUROCONTROL




Figure 11: One of the acknowledgement's parameter is not waited one If the acknowledgement parameter/value number of pairs is not the waited one, error feedbacks are rendered, showing that the pilot does not understand the clearance. The figure 12 shows one of these cases.

Figure 12: The acknowledgement contains more parameter/value pairs than the clearance

3.4. Message inconsistencies feedback If the ASR results are inconsistent – i.e. those results are not possible in the current context – a feedback is rendered on the controller or pilot's part according to the dialog part in which inconsistency is found. The figure 13 shows the feedback on the ASR state area when the recognized acknowledgement callsign does not exist in the current controlled area. figure 14 shows the feedback on an inconsistency discovered in the pilot's response. The watermarking and clearance/acknowledgement plugins are then disabled until the next controller-pilot dialog. This feedback permits to prevent the controller that he can't trust the speech recognition during this exchange.




EUROCONTROL




Figure 13: Inconsistency on the clearance callsign

Figure 14: Recognized acknowledgement's callsign does not exist in the current context If the inconsistency concerns the parameters or the values of the orders while the callsign exists in the current context, only the clearance/acknowledgement match plugin is disabled. The figure 15 is an example of the feedback if the controller pronounces the clearance: “AFR023 accelerate to 0.75 mach” and the ASR understands “AFR023 accelerate to 15 machs”.

Figure 15: The clearance's order is inconsistent




EUROCONTROL




4. Conclusion The demonstrator described in this document allows to prove the efficiency of a system representing the dialog between controller and pilot. Two ASR systems and a graphical interface, representing a radar display and the status of speech recognition, are interacting together, exchanging messages in real time. This demonstrator is a strong basement on which further improvements can be studied.




EUROCONTROL




5. Appendix A: Controller Grammar ; SCOPE Project

; Controller's clearance

; Version : 2004 Nov. 23

; author : ehrette

#include "callsign.grammar"

#include "Number.grammar"

.CClearance [

(

Callsign:s ?ORIGIN good morning

)

(

Callsign:s

?[

(FL_ORDER:i1 ?(and [CAP_ORDER:i2 SPD_ORDER:i3]))

([CAP_ORDER:i2 SPD_ORDER:i3] ?(and FL_ORDER:i1))

(CAP_ORDER:i2 and SPD_ORDER:i3)

(SPD_ORDER:i3 ?(and CAP_ORDER:i2))

]

?[(due traffic)(for spacing)(for delay)]

)

]

{

<callsign-slot $s>

<instruction1 $i1.instruction>

<value1 $i1.value>


<value2 $i2.value>


<value3 $i3.value>

}




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative ;flight level

FL_ORDER [

(?[continue (when ready)] climb to FL_ALT:n)

{return([<instruction climb><value $n>])}

(?[continue (when ready)] descend to FL_ALT:n)

{return([<instruction descend><value $n>])}

(maintain FL_ALT:n ?[(until further advised)(while in controlled airspace)])

{return([<value $n>])}

(expedite climb)

{return([<instruction climb>])}

(expedite descend)

{return([<instruction descend>])}

(stop descend)

{return([<instruction stop-descend>])}

(stop climb)

{return([<instruction stop-climb>])}

]

;cap

CAP_ORDER [

([fly continue] heading Number_360:n)

{return([<instruction heading><value $n>])}

(turn Turn_Dir:d [(heading Number_360:n)(Number_360:n degrees)])

{return([<instruction $d><value $n>])}

(continue present heading)

{return([<instruction continue-heading>])}

(stop turn ?(heading Number_360:n))

{return([<instruction stop_turn><value $n>])}

(make a three sixty turn Turn_Dir:d)

{return([<instruction $d><value 360>])}

]




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative ;speed

SPD_ORDER [

(speed Number_mach:n mach)

{return([<instruction speed-mach><value $n>])}

(speed Number_knots:n knots)

{return([<instruction speed-knots><value $n>])}

(maintain Number_mach:n mach)

{return([<instruction speed-mach><value $n>])}

(maintain Number_knots:n knots)

{return([<instruction speed-knots><value $n>])}

]

; Specific grammars

; -----------------

FL_ALT [

(flight level Number_FL:n){return($n)}

(Number_feet:n feet){return($n)}

]

Number_FL [

seventy {return(70)}

eighty {return(80)}

ninety {return(90)}

(?one hundred) {return(100)}

(?one hundred ?and [Ten:n Twenty_to_ninety:n]) {return(add(100 $n))}

(two hundred) {return(200)}

(two hundred ?and [Ten:n Twenty_to_ninety:n]) {return(add(200 $n))}

(three hundred) {return(300)}

(three hundred ?and [Ten:n Twenty_to_ninety:n]) {return(add(300 $n))}

(four hundred) {return(400)}

(four hundred ?and [Ten:n Twenty_to_ninety:n]) {return(add(400 $n))}

(five hundred) {return(500)}

(five hundred ?and [Ten:n Twenty_to_ninety:n]) {return(add(500 $n))}

]

Number_feet [

(Non_zero:t thousand) {return(mul(1000 $t))}

(Non_zero:t thousand ?and five hundred) {return(add(500 mul(1000 $t)))}

]




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative ; Speed Unit

Sp_Unit [

mach {return(speed-mach)}

knots {return(speed-knots)}

]

; Speed Direction

Sp_Dir [

increase {return(increase-speed)}

reduce {return(reduce-speed)}

]

; Turn Direction

Turn_Dir [

left {return(turn-left)}

right {return(turn-right)}

]

;Control Tower place

ORIGIN [

(paris center)

]

; Restrictions on Numbers

; -----------------------

Number_mach (

Zero point

[

[(seven ?zero) seventy] {return(0.70)}

([seven seventy] five) {return(0.75)}

[(eight ?zero) eighty] {return(0.80)}

([eight eighty] five) {return(0.85)}

[(nine ?zero) ninety] {return(0.90)}

]

)




EUROCONTROL



WP4 – R4 – Demonstrator: Implementation and source code SCOPE – CARE II Innovative Number_knots [

(?one hundred) {return(100)}

(?one hundred ?and Up_to_two:n) {return(add(100 $n))}

(two hundred) {return(200)}

(two hundred ?and Up_to_two:n) {return(add(200 $n))}

(three hundred) {return(300)}

(three hundred ?and Up_to_two:n) {return(add(300 $n))}

]

Number_360 (NumDigit3:c NumDigit:d NumDigit:u)

{return (add(add(mul(100 $c) mul(10 $d)) $u))}

NumDigit3 [

[zero oh] {return(0)}

one {return(1)}

two {return(2)}

three {return(3)}

]




EUROCONTROL




6. Appendix B: Demonstration scenario

CONTROLLER PILOT

callsign fox golf tango oscar delta fox golf tango oscar delta

match November kilo tango climb to flight level

two hundred and forty

November kilo tango climbing to flight

level two hundred and forty

match November romeo speed zero point eight

mach

November romeo speed zero point eight

mach

match Swissair seven zero three turn left heading

one two five

Swissair seven zero three turning left one

two five degrees

match

2 orders

November kilo tango descend one thousand

feet and speed one hundred knots

November kilo tango descending one

thousand feet and speed one hundred knots

mismatch

callsign

November kilo tango climb to flight level

two hundred

November romeo climbing to flight level

three hundred

mismatch

value

Swissair seven zero three turn left heading

one two five

Swissair seven zero three turning left

heading one two six

mismatch

order

Hotel zulu speed one hundred and fifty

knots

Hotel zulu descending to flight level one

hundred and fifty




EUROCONTROL



wp4 – r4 – demonstrator: implementation and source code · 2007-11-12 · wp4 – r4 –...

Documents