d7.32 human machine interface 2 - ipatchproject.eu · ipatch d7.32 4 disclaimer the content of the...
TRANSCRIPT
IPATCH D7.32
1
D7.32 – Human Machine Interface 2
Due date of deliverable: 31/01/2017
Actual submission date: 10/02/2017
Organisation name of lead contractor for this deliverable: BMT
Revision: 1.0
Grant Agreement No: 607567
Project Acronym: IPATCH
Project Title: Intelligent Piracy Avoidance using Threat detection and Countermeasure Heuristics
Funding Scheme: SEC-2013.2.4-2
Start date of project: 01/04/2014
Duration: 36M
Project co-funded by the European Commission within the 7th Framework Programme (2007-2013)
Dissemination Level:
PU Public
RE Restricted to a group specified by the consortium (including the Commission
IPATCH D7.32
2
Table of Contents
Executive Summary ................................................................................................................................. 6
1 Introduction ....................................................................................................................................... 7
1.1 The IPATCH Project ................................................................................................................ 7
1.2 Deliverable Overview ............................................................................................................... 7
2 HMI Requirements ........................................................................................................................... 9
2.1 Requirements .......................................................................................................................... 9
3 Specifications ................................................................................................................................. 11
3.1 System Functions .................................................................................................................. 11
3.2 Layout .................................................................................................................................... 12
3.3 Integration .............................................................................................................................. 13
4 Implementation ............................................................................................................................... 15
4.1 Map View ............................................................................................................................... 15
4.2 Multi-Camera View ................................................................................................................ 17
4.3 Timeline View ........................................................................................................................ 19
4.4 Decision Support View .......................................................................................................... 21
4.5 Electronic Countermeasures Manual .................................................................................... 22
4.6 Alarm Notification .................................................................................................................. 24
4.7 Playback engine .................................................................................................................... 24
4.8 Data Parsing and Processing ................................................................................................ 25
5 Evaluation ....................................................................................................................................... 27
6 Conclusions and future work .......................................................................................................... 28
Appendix A: HMI Requirements Full List ............................................................................................... 29
IPATCH D7.32
3
Document Summary Information
Authors and Contributors
Initials Name Organisation
SK Socrates Katsoulacos BMT
Revision History
Revision Date Initials Description
V0.1 14/12/2016 SK First draft
V0.2 20/01/2016 SK Updated draft
V0.3 06/02/2017 SK Final document for review
V1.0 10/02/2017 TC Reviewed version for submission
Quality Control
Role Name Date
Peer Reviewer Philipp Lohrmann 01/02/2017
Work Package Leader Grzegorz Taberski 06/02/2017
Project Manager Tom Cane 10/02/2017
Security Scrutiny Committee Review
Comments
No security sensitivity issues
Recommended Distribution
Consortium and Commission services
Date 06/02/2017
IPATCH D7.32
4
Disclaimer
The content of the publication herein is the sole responsibility of the publishers and it does not
necessarily represent the views expressed by the European Commission or its services.
While the information contained in the documents is believed to be accurate, the authors(s) or any other
participant in the IPATCH consortium make no warranty of any kind with regard to this material including,
but not limited to the implied warranties of merchantability and fitness for a particular purpose.
Neither the IPATCH Consortium nor any of its members, their officers, employees or agents shall be
responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission
herein.
Without derogating from the generality of the foregoing, neither the IPATCH Consortium nor any of its
members, their officers, employees or agents shall be liable for any direct or indirect or consequential
loss or damage caused by or arising from any information advice or inaccuracy or omission herein.
IPATCH D7.32
5
List of Figures
Figure 1: HMI Layout ............................................................................................................................. 13
Figure 2: Physical Connections between the IPATCH modules ........................................................... 14
Figure 3: Map View - Exploration Mode ................................................................................................ 16
Figure 4: Normal mode - Vessel under attack ....................................................................................... 17
Figure 5: Multi-camera View .................................................................................................................. 17
Figure 6: Multi-camera View with a single camera on Full Screen ....................................................... 18
Figure 7: Timeline View – Slider at start ................................................................................................ 20
Figure 8: Timeline View – Slider in the middle ...................................................................................... 20
Figure 9: Decision Support View ........................................................................................................... 21
Figure 10: Electronic Countermeasures Manual ................................................................................... 23
Figure 11: Electronic Countermeasures Manual - Search for “speed” keyword ................................... 23
Figure 12: Alarm notification (annotated with blue rectangle) ............................................................... 24
Figure 13: Playback Engine GUI ........................................................................................................... 25
List of Tables
Table 1: Track and Object Detections Colour on Map View ................................................................. 15
Table 2: Data received and processed by the HMI ............................................................................... 25
List of Abbreviations
Abbreviation Description
HMI Human Machine Interface
GUI Graphical User Interface
AIS Automatic Identification System
IP Integration Platform
Protobuf Protocol Buffers
RAM Random access memory
kBs Kilobytes
MBs Megabytes
IPATCH D7.32
6
Executive Summary
The IPATCH Human Machine Interface (HMI) is what the seafarers using the system actually see. It
provides several views and graphical interfaces that allow users to visualise the traffic around the vessel
coming from different sources such as the radar, AIS and the on-board cameras, in combination with
the output of the algorithms. The aim is to highlight any detected potential suspicious behaviour against
the vessel and inform the users as early as possible.
In particular, users can watch the traffic around the vessel in a map view which provides a bird’s eye
view of the current situation. Suspicious behaviour will be easily identifiable on this view by marking the
potential intruder boats with a red colour. In addition, the user will have the ability to switch between
layers of information to see on the screen, for example a combination of radar and fused/merged tracks
but not AIS and vice-versa. Furthermore, drill-down capabilities are implemented in the map-view in the
form of mouse hovering over one of the object detections displayed, providing more detailed information
about the object in question. In case the system detects suspicious behaviour, it will activate a red
flashing window at the top of the main screen which will display information regarding the threat.
Moreover, the live video stream with object detections and their annotated tracks will be accessible to
the seafarers at any time on a video view providing the ability to see up to four cameras simultaneously.
In addition to that, a timeline view is provided in which all the suspicious events that were detected are
displayed in chronological order. With the help of a slider the users can go backwards and forwards in
time (up to the last threat detected) to inspect the situational picture that raised an alarm in the system.
Finally, the HMI provides an electronic version of the Countermeasures Manual, as well as suggestions
regarding the current situation, generated in real time by the Decision Support Tool (see D7.2).
IPATCH D7.32
7
1 Introduction
1.1 The IPATCH Project
Funded by the European 7th Framework Programme, the IPATCH project addresses Security Topic
SEC-2013.2.4-2: Non-military protection measures for merchant shipping against piracy. The goal of
the IPATCH project is three-fold:
1. To perform an in-depth analysis of the legal, ethical, economic and societal implications of
existing counter piracy measures.
2. To produce well-founded recommendations to the industry in the form of a manual, extending
and complementing the Best Management Practices document and to support the use and
further development of countermeasures.
3. To develop an on-board automated surveillance and decision support system providing early
detection and classification of piracy threats and supporting the captain and crew in selecting
the most appropriate countermeasures against a given piracy threat.
The analysis performed under (1) will lead to recommendations for the use of countermeasures in a
range of scenarios, structured as a manual (2), and development and implementation of a proactive
surveillance system forming part of the system developed in (3). The situational awareness system will
robustly monitor the area around maritime vessels, providing early warning to crew members if piracy
threats are detected. A low false alarm rate due to environmental or other innocuous events, combined
with high threat detection sensitivity are central ambitions of the project.
To achieve these goals, a multispectral sensor suite comprising both passive and active sensors is
envisaged, i.e., a system based on radar, visual and thermal sensors. The sensor suite will be
complemented with advanced algorithms for information fusion, object detection and classification, and
high level modelling of intent and behaviour analysis. The IPATCH project is strongly user-driven and
demonstration of the developed surveillance system will be conducted in two different maritime
environments.
1.2 Deliverable Overview
The IPATCH Human Machine Interface (HMI) is one of the main development components of the
IPATCH project. There are two main reasons that the HMI is such an important and integral component
of this project: a) it allows the end users to see in a user-friendly format the output of the system, and b)
it allows the developers of the algorithms used in the project to test and optimize them. Finally, using
the HMI we can show proof of concept by demonstrating playbacks of real life scenarios and the outputs
of the algorithms regarding these scenarios in a smart and user-friendly way for a non-expert.
The scope of this deliverable is to give an overview of the HMI. This includes technical characteristics,
its main functions based on the user requirements, the layout of the system as well as the graphical
user interfaces (GUIs) it provides and finally the capabilities of connecting to external modules to
exchange information through the Integration Platform (IP) and Protocol Buffers (Protobuf).
The contribution of this deliverable to project goals and objectives is to demonstrate the capabilities of
the HMI showing that it is a user-friendly and intuitive system to use especially in stressful situations
such as a pirate attack when decisions have to be taken quickly.
IPATCH D7.32
8
This deliverable follows the same structure as the first deliverable for the Human Machine Interface. We
visit everything that was discussed in the previous deliverable and update it to match the current
implementation status of the HMI. In addition to that, we have introduced some more sections that
describe new parts and mechanisms of the HMI as well as challenges and limitations.
The deliverable follows the structure below:
Section 2 - Requirements: The requirements section describes the main requirements that we have
considered for designing and implementing the IPATCH HMI.
Section 3 - Specifications: The specifications section describes the different categories of functions
that we have derived from the requirements. In addition, we explain the look and feel of the IPATCH
HMI on the layout section. Finally, we discuss briefly the integration of the HMI with the rest of the
IPATCH system.
Section 4 - Implementation: The implementation section provides images of the current work that has
been done on the HMI and explanations of these images which reflect the functionality that the HMI
provides to the user.
Section 5 – Evaluation: The evaluation section provides an insight into how the HMI performs under
load and its limitations.
Section 6 – Conclusion: In the conclusion section we wrap up the document and we discuss future
work.
IPATCH D7.32
9
2 HMI Requirements
There are three important reasons why a HMI must be included in the development of the IPATCH
system:
1. The HMI is what the users of the system will “see” - it is the main means of interacting with the
system. For the demonstrator cases, the HMI plays an important role because the end-users
will not be data fusion experts. Their feedback on the project outputs will be largely based on
what they experience through the HMI.
2. During the project, the HMI should support the development and testing of the different
algorithms to be developed in WP 5, 6 and 7. Individual algorithms can be developed in “lab
conditions” but this will become more difficult when they are linked together as part of an
integrated system. The HMI provides a way of monitoring and assessing the behaviour of the
system as whole, querying and visualising inputs/outputs for debugging, running tests and so
on.
3. Finally, the HMI could provide a very useful tool for configuring the IPATCH system for use on
different platforms, including tools for adding and removing different sensor and algorithm
capabilities.
2.1 Requirements
A list of requirements for the IPATCH system was identified in Deliverable 2.1 – see Appendix A for the
full list of the requirements related to the HMI. The following requirements are relevant to the HMI
(ranked in order of priority given by the end users):
Early warning notification;
Provide simple information from complex situation and multiple sensors to allow people under
pressure to make the best decisions;
Provide countermeasure suggestions and efficient browsing of the countermeasures manual;
User friendliness and provision of information on a “need to know” basis.
As the above list is very high-level, we have added a number of additional requirements to guide the
specification and development of the system. In addition, we have received feedback from the end users
in the project on what is important and how to make data visualization and threat awareness more
succinct for the user. We have also included information about the feedback on the list below.
The HMI should provide an intuitive graphical overview of the current situation.
A map view and camera views with annotations provides the most succinct way for representing
the relevant data, as recommended by the stakeholders.
The user should receive an alarm notification as early as possible with only information that is
absolutely necessary.
From the stakeholders’ comments that have been gathered, the best way to describe a threat
from a seafarer’s point of view it to refer to the suspicious targets using angle measurements
and their speed and distance from the vessel.
IPATCH D7.32
10
The HMI should provide easy to understand and effective countermeasure suggestions in case
of a threat.
The HMI should have the capability to efficiently browse the countermeasures manual.
The HMI should display data from all the sensors (cameras, radar and AIS) in an easy to
understand way, including the output from the algorithms.
As mentioned above, from the feedback received from the stakeholders, the best way to
describe visually such data is on a map view and by annotating the camera views.
The system should provide real-time video feeds.
IPATCH D7.32
11
3 Specifications
3.1 System Functions
The HMI must have functionality which supports the activities of the end user. We can divide these
functions into separate categories: “Situation monitoring”, “Countermeasures Manual and Real-time
information from the decision support tool”, “Reporting” and “System Access & Administration”.
1. Situation Monitoring
The HMI should provide the means for continuous monitoring of the area around the vessel.
This includes displaying information taken from the radar and AIS on board of the vessel as well
as the mounted cameras. Furthermore, it should give a notification/alarm to the users as early
as possible if a suspicious behaviour has been detected, displaying clearly both on the camera
view and the map view the threat that has been detected. Finally, various controls such as
buttons and check boxes, and mouse hovering (for drill-down capabilities) should allow the user
to see various information regarding the current situation and the theat.
2. Countermeasures Manual and Real-time information from the Decision Support Tool
The HMI should provide an interface to the electronic version of the Countermeasures Manual
which should allow the user to find the information they are looking for in the fastest and most
efficient way possible. In addition to this, the HMI system should display real-time information
received from the Decision Support Tool in order to facilitate decision making during an attack
as well as provide suggestions on the best course of action.
3. Reporting
The HMI should provide reporting capabilities that allow users to fill in electronic forms on the
HMI which will then automatically send them to the relevant authorities. These forms need to
be filled in by the captain of the vessel in the event of an attack and will capture all the important
information regarding it.
4. System Access & Administration
The HMI should provide capabilities that allow different kinds of users to access the system.
This includes both the technical personnel that are responsible for the maintenance of the HMI
as well as the vessel personnel. The HMI should provide different access types for users
depending on what their role is and forbidding or allowing access to different information and
views of the system depending on this role. In order to achieve this, the system should provide
user accounts and encrypt its data. In addition, the HMI should provide administration
capabilities such as configuring and optimization of other modules of the IPATCH system, for
example the cameras.
IPATCH D7.32
12
3.2 Layout
The layout of the HMI follows a minimalistic design that consists of the minimum number of buttons
possible, as well as the least possible number of views available to the user. By this, we are trying to
create an easy-to-use and intuitive system and reduce the workload during the use of the HMI. On each
view, we provide only the absolutely necessary information that the user needs depending on the
situation.
Because of the nature of a ship’s bridge environment, the HMI is designed for use on a touch-screen
monitor. This step also grants more freedom to the user and will allow them to have a more engaging
and user-friendly GUI.
Below we can see an image of the HMI layout. The four buttons on top are giving access to the main
views that the HMI provides to the users. Starting from the left, the first buttons opens the multi-camera
view, the second one opens the map-view, the third one opens the timeline view and the final one opens
the countermeasures recommendations information.
The views, when opened, are displayed in the middle of the screen below the four buttons and between
the Controls, and the Vessel Info and Recommendations areas. If you open a view and then another,
the second view will be displayed instead of the first one. Furthermore, on the left hand side of the HMI
there is a “Controls” area which when one of the views discussed above is displayed, will contain control
buttons and options relevant to that view.
On the right hand side, there is a “Vessel Info” area which will display in real time all the vessel’s current
information such as the speed, latitude and longitude, speed over ground, etc. Below that, there is a
“Recommendations” area which will display in real-time the suggested countermeasures to be used in
the current situation and which also includes a button that if pressed provides immediate access to the
electronic countermeasures manual. The countermeasures on the “Recommendations” area are colour-
coded using a gradient of colours starting with green (meaning “highly effective”) to yellow (“moderately
effective”) and orange (less than moderately effective). The effectiveness of the countermeasures
suggested is computed by the Decision Support Tool and takes into account various factor – please
refer to the Decision Support Tool document – D7.2.
Finally, the user has the flexibility to change position on the screen of the panel that contains the top
four buttons area, as well as the controls, vessel info and recommendations area each one individually.
The top four buttons area can be docked at the bottom of the screen as well as at the top, which is its
initial position, whereas the others can be docked on any of the four sides of the screen. In addition, all
of the areas can be kept as independent windows (i.e. not docked on the main screen but existing as
separate views).
IPATCH D7.32
13
Figure 1: HMI Layout
3.3 Integration
The IPATCH system consists of various modules, each one playing a vital role in the overall system.
Some of these modules are responsible for the object detection or the situation assessment while others
are responsible for calculating the best possible countermeasures to be used in a certain situation
(Decision Support Tool – D7.2). All of these modules need to exchange information, and this is facilitated
with the use of the Integration Platform (IP) and the Protocol Buffer (Protobuf).
The HMI is no exception to this, since it requires all the information calculated by the other modules in
order to parse them and display them in a user-friendly format. Therefore, in addition to the above
functional requirements, the HMI must be able to connect to the IP and the Protobuf services to
exchange information with all the other modules of the IPATCH system.
The Protocol Buffer1 is a mechanism for serializing structured data. It allows the developers to define
how they want their data to be structured and then use special generated source code to easily write
and read the structured data to and from a variety of data streams using different programming
languages. The use of Protobuf in the IPATCH system alleviates the network load on the IP and data
repository. In some cases, this is especially true because of the high volume of data that have to be
exchanged between the IPATCH modules.
1 Protocol Buffer Developers Guide – https://developers.google.com/protocol-buffers/docs/overview
IPATCH D7.32
14
The diagram below shows the communication links between the modules of the IPATCH system,
including the HMI. Using these communication links, the HMI can receive data from sensors such as
the AIS, radar and cameras. In addition, it can receive data produced by the detection and tracking
algorithms in the Object Detection and Situation Assessment Modules, as well as countermeasure
suggestions from the Decision Support Tool. All the data is filtered by the HMI in order to display it to
the user at the right time in an intuitive and user-friendly format. Furthermore, the HMI can query the
Countermeasures database (CMs DB) and the Geospatial database. The former contains the
information to be used on the electronic version of the countermeasures manual while the latter contains
geospatial data that will be used in the “Map” View. Finally, the HMI is connected to a map tiles server
(see Section 4.1 for more information) which runs locally in order for the HMI to request tiles for
displaying a map on which the received detections will be shown.
Figure 2: Physical Connections between the IPATCH modules
IPATCH D7.32
15
4 Implementation
4.1 Map View
The map view can be opened by clicking the second button from the left on the top of the HMI window
(annotated with a blue box on the Figure 3: Map View - Exploration Mode). The map displayed on the
map view has been updated in the latest version of the HMI to a very realistic open licensed map named
OpenStreetMap2. This map provides a very detailed representation of the world with continuous updates
which are easily deployable on the HMI. In order for the map to be displayed on the HMI, a tile server
has been deployed locally on the HMI machine. The purpose of the tile server is to host the map as a
web-service in order for the HMI to request tiles that contain a representation of the world.
While the map view is displayed, the user can observe the traffic around the vessel, which is being
detected by all the sensors on the ship (i.e. AIS, radar and cameras), as well as the current position of
the ship. The controls area on the left of the screen allows the user to enable/disable what is being
shown on the map area. The options for the tracks are AIS, Radar and Merged Tracks which will display
the relevant tracks on the map if their check boxes are ticked. The object detections are displayed as
dots on the map, while the tracks are the trajectories of these object detections shown as lines between
object detections. In addition, an important feature of the tracks is their length which represents the
speed with which the object is moving, i.e. a small track line denotes a slow speed, a large track line
denotes that the detected object moves at high speed, and finally no track line shows that the object
detected is not moving. This feature applies to all tracks shown on the map view – AIS, radar, merged
tracks and our own vessel track. An example of this feature can be seen in Figure 4: Normal mode -
Vessel under attack, in which we can observe that the lines of the skiffs are more than two times larger
than the one of our vessel, which indicates to the user that the skiffs move at more than twice the speed
of our vessel.
The tracks and object detections on the map view are colour-coded to help the user to identify the source
of the track/object detection and whether they pose a threat to the on board vessel. The table below
describes the relation between the source of the track and object detections, and the display colour.
Table 1: Track and Object Detections Colour on Map View
Source Colour
Radar Orange
AIS Dark Blue
Merged Track Green or Red
IMU (our vessel) Black
As shown in the table above, the merged tracks are the only ones that can be denoted with two different
colours and that is to provide to the user an indication of whether the relevant object detection(s) are
regarded as suspicious or not – red colour for suspicious behaviour and green for normal behaviour
(see Figure 3 and Figure 4).
2 OpenStreetMap – https://www.openstreetmap.org
IPATCH D7.32
16
Our vessel is represented on the map with an icon that has a light blue circle with a white arrow in it.
The position of the icon is updated in real-time to denote the real position of our vessel on the globe and
the arrow is showing the heading of the vessel. The rest of the information, including the information
mentioned here regarding our vessel, can be seen at all times on the right of the screen in the “Vessel
Info” area.
Another important feature of the map view is the drill-down capabilities that it provides to the user with
the ability to hover over an object detection (circle) in order to see information regarding that detection.
When the user hovers over a circle, a white box will be displayed on the top left of the map displaying
information such as navigational information (latitude, longitude, course over ground, etc.) as well as
the current behaviour of the selected detection (Figure 4: Normal mode - Vessel under attack).
The map view has two modes, an “Exploration” (Figure 3: Map View - Exploration Mode) and a “Normal”
(Figure 4: Normal mode - Vessel under attack) mode. The user can switch between the two modes from
the Map Controls on the left bottom of the screen, by clicking the Normal mode button when the map is
in Exploration mode or clicking the Exploration mode when the map is in Normal mode – the currently
active mode is shown above the mode switching button. In the Exploration mode, the user can pan on
the map and move around to see any part of the world, while in the Normal mode the panning
functionality is disabled and our vessel is always displayed in the middle of the map.
Finally, we note that the map has zooming and panning capabilities for both a normal monitor as well
as a touch screen in which the user can use pinching and panning finger gestures respectively.
Figure 3: Map View - Exploration Mode
IPATCH D7.32
17
Figure 4: Normal mode - Vessel under attack
4.2 Multi-Camera View
Figure 5: Multi-camera View
IPATCH D7.32
18
The above image shows the multi-camera view which can be opened by clicking the first button from
the left on the main HMI screen (annotated with a blue box in Figure 5: Multi-camera View). This view
allows the user to see a live feed from up to four cameras at the same time. In addition, object detections
(rectangles around the detection) and their tracks (lines) are being displayed on each of the four camera
areas, if there are detections associated with it. Red colour object detections denote a suspicious
detection, while green means normal behaviour. The cameras can be either visible light cameras or
infrared cameras mounted on the vessel. The user can select the camera which they would like to see
on any of the four camera areas by selecting it from the drop down box below each camera area.
A feature of this view is to allow the user to maximize any of the four displayed cameras in order to see
the camera stream of the selected camera in full screen. This is achieved by clicking on the “Expand”
button under the desired camera feed. While in a full screen camera display, the user can select any
camera from the scroll down menu to change which camera feed is being show in full screen. An
example of the full screen display for a camera feed is shown in Figure 6: Multi-camera View with a
single camera on Full Screen.
An important feature which is currently under implementation is that the multi-camera view will maximize
automatically the camera feed in which suspicious behaviour is detected when an alarm is triggered on
the system.
Finally, on the left under the controls area there are three buttons, “Pause”, “Rewind” and “Go live!”.
These buttons, together with a text that displays the current timestamp, serve the purpose of pausing
and rewinding the camera feeds or going back to live stream respectively. However, currently they are
placeholders and this functionality is scheduled for future work.
Figure 6: Multi-camera View with a single camera on Full Screen
IPATCH D7.32
19
4.3 Timeline View
The timeline view can be opened by clicking the second button from the right on the top of the HMI main
window (annotated with a blue box on the Figure 7: Timeline View – Slider at start). This view allows the
user to see the exact moment when a suspicious event was detected by the IPATCH system. As can
be seen in the figures below, the view contains a map and a slider.
The map displays a snapshot of the situational picture (as a static image) which caused a threat to be
detected at a specific time (when the threat was first detected, and then the latest information if an
update has been received from the threat detection module after the initial threat generation). The map
will display all the involved actors as object detections and their tracks. The main actor (the one that
generated the threat in first place) is annotated in red colour. The other actors involved in the threat are
annotated in yellow colour. In addition, the position of our vessel is displayed with its track and its
heading is shown with a white arrow inside the blue circle (as described on the Map View section - 4.1).
The user can zoom in/out and pan on the map.
Above the map, the timeline view has a slider that can be used to go forwards and backwards
chronologically to see all the suspicious events that the system has detected. Moving the slider will
update the contents of the map i.e. the actors involved in the threat (e.g. moving the slider from left to
right we can see to the threats generated at the same time or after the currently selected threat).
Above the slider, the user can see three timestamps. The two timestamps on the sides of the space
above the slider (annotated with a green rectangle) display the first threat detected (left green rectangle)
and the last threat detected (right green rectangle). The timestamp in the middle (annotated with the red
rectangle) represents the time of the currently selected threat – i.e. the time of the threat currently shown
on the map below the slider.
Finally, the view shows the elapsed time since the threat was detected in seconds (see orange
annotations on Figure 7: Timeline View – Slider at start). For example, on the image below we can that
the last threat detected was 17 seconds ago.
IPATCH D7.32
20
Figure 7: Timeline View – Slider at start
Figure 8: Timeline View – Slider in the middle
IPATCH D7.32
21
4.4 Decision Support View
Figure 9: Decision Support View
The Decision Support View (Figure 9: Decision Support View, above) can be opened by clicking on the
first button from the right on the top of the HMI main window (annotated with a blue box in the figure
above). This view allows the user to see information about the real-time recommendations on which
countermeasures to use in the current situation, generated by the Decision Support System. These
suggestions are calculated in real-time by the decision support tool (D7.2) and are then sent via the
integration platform to the HMI. The suggestions are displayed on the right of the screen under
“Recommendations” at all times, independently of which main view is open (more information about the
Recommendations can be found in Section 3.2 Layout).
In addition, if the user needs to quickly see more information about the suggested options, such as the
minimum and maximum range, then he/she can quickly look at the Decision Support View. This view
contains a table with the most important characteristics of the countermeasures, and uses the same
colour coding that was used on the “Recommendations” at the right of the screen. Furthermore, all the
columns of the table can be sorted appropriately. This view is still under development and will be
finalised soon based on feedback from the Stakeholder Meeting in February 2017.
IPATCH D7.32
22
4.5 Electronic Countermeasures Manual
The electronic version of the countermeasures manual (Figure 10: Electronic Countermeasures Manual)
can be opened through the Decision Support View (as explained above) or through the
Recommendations area which is located on the right of the HMI at all times, providing an “Open CMs
Manual” button. The HMI is using a local database to request the countermeasures information.
The countermeasures manual consists of three main parts. The top part contains a search functionality
area and a table. The table contains a list of all the countermeasures names and descriptions that exist
in the countermeasures database. By clicking on one of the column headings of the table, the relevant
column data are ordered alphabetically. Once the user selects a row in the table, the other two parts
described below are updated accordingly. Finally, the user can type key words in the input area on the
search area and click the “Go” buttons in order to filter the table contents (example in Figure 11:
Electronic Countermeasures Manual - Search for “speed” keyword). When the user is done with
searching specific countermeasures and would like to see all the countermeasures again, he/she can
click the “Clear” button, which will also clear the search input area.
The second part of the e-manual consists of the Services and Devices section in which the user can
find information about the services and the devices such as characterisation criteria (countermeasure
time, expected effects and legal and ethical characterisations) as well as rules for implementation and
usage, constraints and examples. This view contains tabs which the user can click in order to select
different services and devices of the selected practice and procedure.
The third and final part of this view displays information on elementary practices. The user can choose
between different practices using the tabs provided and he/she will be shown a description as well as a
table that shows effectiveness of the practice depending on the situation in which it is used. Finally, on
the right side of the view, an image of the selected countermeasure is displayed (if one exists).
We note that the Electronic Countermeasures Manual is opened as a separate window rather than being
displayed in the middle of the main HMI screen, so that it can be used separately or at the same time
with any other view of the system.
IPATCH D7.32
23
Figure 10: Electronic Countermeasures Manual
Figure 11: Electronic Countermeasures Manual - Search for “speed” keyword
IPATCH D7.32
24
4.6 Alarm Notification
Figure 12: Alarm notification (annotated with blue rectangle)
The image above shows an example of an alarm (annotated in the blue rectangle) when a threat is
detected by the system. The alarm notification has been redesigned from the first version of the HMI to
be part of the main HMI screen rather than a pop-up window. The redesigning of the alarm from a pop-
up to a notification was done to reduce the number of windows shown concurrently to the user since the
pop-up was generated as a separate window. Now the alarm is part of the main screen of the HMI and
is displayed without obstructing any of the views of the HMI when it is active.
When an alarm is triggered, an area below the main four buttons at the top of the HMI will be flashing
red to get the attention of the user, regardless of which view the user is currently in. The notification
contains a description of the detected threat as well as relevant information such as the speed of the
detected object, its angle from the magnetic north and the distance from our vessel. Finally, there is an
“Acknowledge” button on the bottom left of the notification area which allows the user to silence the
alarm.
4.7 Playback engine
For testing purposes, we have implemented a playback engine on the HMI, which is only available to
the developers and not the end users. This engine has a minimal GUI (Figure 13: Playback Engine GUI)
which allows the developer to pause and start the playback sequence as they wish. The playback engine
provides the capability to simulate a scenario as run in real-time.
More specifically, when the playback engine runs it emits signals to the Multi-camera, Map and Timeline
views and once these views receive the signal, they fetch the required data from a cache space and
display them in the same format as described above in Section 4 Implementation. The cached data are
IPATCH D7.32
25
read using parsers that have been implemented for this purpose and are stored in a special data
structure created for the purposes of testing the HMI. The caching of the data (images, AIS, radar, IMU,
etc.) has to be completed before the playback engine runs, with the exception of images which will be
cached up to a number and continue to be cached while the playback engine runs (described in more
details in Section 5 Evaluation).
Figure 13: Playback Engine GUI
4.8 Data Parsing and Processing
In order for the HMI to receive all the data that are produced by the various modules of the IPATCH
system, it needs to be able to communicate with them. As we have mentioned above in Section 3.3
Integration, the HMI uses two different ways to achieve this. The first one is through the Integration
Platform and the second one is via a Protocol Buffer stream. In addition, another step after receiving the
required data is the processing and parsing that is needed to be able to logically relate them to each
other, as well as displaying them at the right time (when a threat with confidence value higher than a
predefined threshold - set by the developers - is received) and in the right format for the user.
The following types of data are being received and processed by the HMI:
Table 2: Data received and processed by the HMI
Type Medium Type Medium
Object Detection Protobuf Threat IP
Images Protobuf Merged Track IP
AIS detection Protobuf Relation IP
Radar detections Protobuf Behaviour IP
IMU Protobuf Actor IP
Track trajectories Protobuf Common Operational Picture IP
Recommendation IP
IPATCH D7.32
26
From the HMI point of view, to receive the above data, it is required to run multiple threads, with each
one of them listening to incoming messages from the IP and the Protobuf services. Once the data have
been received, they are parsed with the implemented parsing functions and are kept in special high
search speed data structures called “Hash maps” in order to be accessed quickly when needed.
While data is continuously sent to the HMI, it does not mean that they are ready to be displayed to the
user immediately. However, this is being decided in real-time by the mechanisms of the HMI depending
on what data are already stored and what data were just received. The decision depends on whether
releasing data to the user will show a complete picture of a threat without any vital information missing.
For example, since each IPATCH modules runs separately, the merged tracks of an actor of a threat
will be received earlier than the actual threat since the computation of a threat takes longer to process
on the relative module. For this reason, the merged tracks will be cached internally on the HMI and when
the threat arrives, the referenced suspicious merged track on the threat will be found from the cached
data and their information will be displayed to the user as a suspicious merged track, which in this case
is to make the track red colour. When a threat is received, the HMI will trigger the relevant events to the
views described in Section 4 Implementation, resulting in the processing of the data so that it can be
transformed into a user friendly format and finally be released to the user via the views.
IPATCH D7.32
27
5 Evaluation
This section describes the technical performance of the HMI and some specific challenges which are
faced due to the nature of the HMI as the final recipient of all the data of the IPATCH system.
One of the main challenges for the HMI development was the use of a realistic map in order for the user
to be able to see an up-to-date high quality bird’s eye view of the current situation around the vessel. In
order to overcome the problem of providing a realistic representation of the world, we had to deploy a
“Map Tile Server”. This server runs locally on the same machine as the HMI and provides map tiles
when requested by the HMI in order to make a representation of the map. However, there is a limitation
with this approach. In order to have a representation of the whole globe or even just Europe, the server
that runs the tile server service (currently localhost) requires to have very high memory space (RAM)
and high processor requirements due to the large volume of data that has to processed and served the
requested client. If these hardware requirements are not met, the user of the HMI can experience
significant lag (of many seconds) until the map is drawn in the actual map view, making the map view
unusable for significant amounts of time. To overcome this problem we currently deploy the tile server
with only the map of specific countries that the trials take place in (e.g. France). This almost solves the
problem, with very little lag happening and only when the user zooms in a lot (which is not currently
allowed by the implementation for this exact reason). A solution as a whole would be to have a separate
machine with the hardware requirements needed in order to run the tile server as a standalone server
for the purpose of the HMI-Map Tile Server requests.
Another challenge during the development of the HMI was the buffering of thousands of high quality
images in order to achieve playback capabilities for testing purposes, which can also be a problem for
feature development of rewind capabilities for the multi-camera widget. Due to the quality of those
images, their size has been between 500 kBs – 1MB. For this reason, it is not feasible to load all the
images for all the cameras into the RAM and just replay them. The solution to this was to create some
special functions that will only cache the first few hundreds of images for each camera, and while the
replay is running, separate threads will remove the images that have been replayed from the cache
space and immediately load into the cache the ones that have to come next. This means the RAM of
the computer does not get full, which allows the HMI to use the memory needed for other intensive
processes. However, there is a limitation to this method as well. The thread that loads an image to be
replayed most of the time runs slower than the playback tick clock. This is because the images are of
high quality so their loading takes more time than the timer clock that ticks every few milliseconds – as
low as 100ms tick interval. This means that with large sequences of data, there might be a point in time
when there are no images loaded for replay yet since the playback has to catch up with the loading
thread. The main solution we used for this was the use of more powerful hardware (a gaming computer
more specifically because of its very good computation speeds), as well as optimizing the multithreading
mechanisms of the playback engine as much as possible. Note that this challenge is only related to the
playback engine, which is used only by the developers for testing purposes.
An important aspect of the HMI is the receipt of the data that is being exchanged between the modules
via the IP and the Protobuf services. We are interested in how the HMI reacts when images from the
camera are coming in and at the same time data are received from the other modules. In the integration
meetings that have taken place during the project, the HMI was tested to observe if there is a lag
between the time that images or data are received and the time at which they are displayed. While
having the HMI connected and receiving all the data (see Section 4.8) and images from up to four
cameras at the same time there was no noticeable lag.
IPATCH D7.32
28
6 Conclusions and future work
In conclusion, the IPATCH HMI is mainly focusing on providing a user-friendly Human Machine Interface
for the end users (i.e. Captains and crews) which will allow them to see the output of the surveillance
algorithms in an understandable way. At the same time, the objective is to minimize the workload and
burden on the operators by showing only the necessary information needed depending on the situation.
Moreover, the HMI supports the algorithm developers in testing their algorithms and demonstrating proof
of concept at the end of the project.
The HMI provides various views to the user that display potential threats to the vessel and its crew, such
as the timeline, multi-camera and the map views. All of these show detected suspicious behaviour by
marking the object in question with red colour on the screen and displaying relevant information, such
as description of the detected threat. In addition, the HMI provides real-time information regarding the
vessel, such as position and speed, allowing the crew to look only to one screen instead of having to
switch between the HMI and on board instruments for seeing this type of information. Another category
of views provides real-time suggestions on which countermeasures to be used and how, depending on
the situation, as well as an electronic form of the countermeasures manual. In addition, the HMI
communicates with the rest of the IPATCH modules via the Integration Platform and the Protocol Buffer
services in order to achieve all of the functions mentioned. Finally, we have implemented a Playback
engine that allows the developers of the HMI to test it by running real-time simulations of scenarios.
During the final months of the project, the functionalities of the HMI will be further refined and finalised.
In the future, the capabilities of the HMI can be further extended regarding security and administration
features, reporting features as well as rewind options. More specifically, for the security and
administration, a completed product should have various user privileges depending on the role of the
user and all data should be encrypted and safely stored, as well as the configuration of the sensors
(such as the cameras) and the algorithms. For the reporting features, the crew will be able to fill
electronic forms on the HMI which will then be automatically sent to the relevant authorities. These forms
need to be filled in by the captain of the vessel in the event of an attack and will capture all the important
information regarding it. Finally, rewind capabilities will let the crew rewind and pause the cameras feed
and the map view in order to observe what has happened in the past in case something was missed.
IPATCH D7.32
29
Appendix A: HMI Requirements Full List
The list below contains the requirements that are relevant to the HMI from D2.1-User Requirements Analysis.
We should note that P1 is high priority for a commercial product (to be acceptable by the users), but in some cases, the functionality is not critical for the
concept prototype used in the demonstration.
Category Ref. D2.1 Page
D2.1 Req. Text Assessment
Criteria Priority Req. met
S_F1 Cover area around the
vessel (sensors) S_F1_R1 67
Expected range of the system: The end-users answers to the question of range provides intervals
for the monitors area When considering maximum range, the answer was: ‘As far as possible’ (while remaining aware
that of the limited range of actual sensors) When considering lower range for pirates’
recognition The requirement was expressed in delay: ‘more than three minutes’ on a high speed
skiff. parameters: speeds, trajectories of pirates vessels
Time delay to reach the
protected vessel P1
Map and Multi-camera views
shows estimate on how far away
pirates are and how quickly they are approaching
S_F1_R2 67
The covered area could be divided in concentric areas, where the specified pirates recognition probability , depending on the capacities of the
selected sensors will be increasing A higher probability may be required in the area
near the vessel ( protection area) parameters: speeds, trajectories of typical pirates
vessels (§3.1.2) Detection range SHOULD be up to the horizon
(and beyond if possible) Sensors MUST cover the sides and stern of the
vessel Sensors COULD cover the front of the vessel
Concentric areas and expected performance
P1
Not directly related to the HMI but more with the
sensors. However, the HMI shows
both on the map view and the multi-camera view what
the current situation is around
the vessel
IPATCH D7.32
30
S_F1_R5 68
Vessel movement: Displacement and yaw, pitch, roll (provided by GPS and by inertial units) should
be considered to assess the geographical coverage area
Parameters: Areas covered by single or fused sensors, precision of GPS and inertial data.
Geographical location of the covered area
Precisions
P2
HMI displays such information
(position, speed, etc.) on the Vessel
Info area
S_F1_R9 69 AIS and radar SHOULD be connected to the
system to provide sensor input
AIS and radar should be
connected to the system
P1
This is achieved via the Protobuf
services which link to the bridge
adapter module
S_F2 Detect and Track targets surrounding
vessels as they progress and
evolve: up-date a local situation
awareness
S_F2_R6 72
Track possible threats as they progress and evolve in order to allow graduated countermeasures
activation, and to follow the effect of these activation on the threat
Track continuity, precision
No perturbation with
countermeasures
P1
Map view and Multi-camera view
display object detections and tracks detected
around the vessel
S_F2_R7 72 System SHOULD be able to track threat over a
period of several hours Duration of
tracking period P?
The HMI shows in red the tracks and object detections
for as long as they are suspicious
IPATCH D7.32
31
S_F3 Recognise threat and Give early warning to the captain and then the crew in case of pirates’
attack
S_F3_R5 74 Early warning: “As much as possible” – must
improve on current situation
Alarm notification as soon as a threat
message is received by the corresponding
module
S_F4 Acquisition of information on
pirates and emission on
reports
S_F4_R1 75
The system should provide images on the pirate ship, allowing visual analysis enabling access to
information as category of the ship, IMO registration, flag, on board equipment (weapons,
ladders…) The resolution should be enough for proof or for
intelligence
Number of pixels on the targets , level of details
P1
The HMI can display the camera
feed from the cameras on the
multi-camera view
S_F4_R2 75
The system should provide images on a potential mother ship:
Identified as Skiff trajectory launching point or as AIS non responding vessel or Information on
geolocalised position of a potential mother ship (List of stolen ships)
System SHOULD be able to detect presence of skiffs on potential mother ships (normally dhows =
fishing vessels) A small field of view EO sensor can be pointed
towards a detected spot
Number of pixels on the targets , level of details
P1 Same as above
IPATCH D7.32
32
S_F4_R4 75 System SHOULD enable security specialists to
add information to the knowledgebase Presence of the
functionality P1
This is not currently
implemented in a way that users can
add information through the HMI
but it can be done manually by
directly accessing the
knowledgebase.
S_F4_R5 75 System SHOULD enable security specialists to
generate reports Presence of the
functionality P1
The HMI at the moment does
use/process data about threats that
are being displayed to the
user but it does not have an automatic way to generate a
report for the security
specialists.
IPATCH D7.32
33
S_F6 Allow interaction with
human operators S_F6_R1 80
The system should present the Local maritime situation, current tracks created on surrounded vessels, their identification, highlighting non AIS
ships, threatening trajectories… This situation could be visualised in a vessel reference or in geographical reference (area
restricted to local situation or area extended to part of the global maritime situation)
Parameters: areas to be visualised, number of ‘friend’ tracks, number of suspicious tracks
Captain acceptance
P1
Map and multi-camera views
show the current situation around the vessel via
camera feeds and their annotations,
as well as annotations on the
map view
S_F6_R2 80
Alerts generation on piracy events: The system should have to avoid false alert and should limit the number of alerts on one event, but the level of alert
could change as piracy behaviour became threatening.
When validated by the captain, the alert could be used to activate on board alarms and
countermeasures, but should also be send ( automatically or with captain acknowledgement)
Parameters: the complexity of the situation ,number of surrounding ships, number and
behaviour of pirates ships
Pertinence, delay of the alerts
P1
The HMI filters internally as much
as possible the Threats received in order to avoid false
alarms. This happens in the
data parsing and processing stage.
S_F6_R3 80 The system should be able to visualise the
movement of the pirates when on board , overlaid on a CAD model vessel
Captain acceptance
P3
Not implemented. We do not have the means (such
as sensors) to calculate the movement of
pirates when on board.
IPATCH D7.32
34
S_F6_R4 80 The system must have to be easy use
Parameters: size and number of the screens, ergonomic design (captain requirements)…
Captain acceptance
P1
Use of a minimalistic design on the HMI. Small number of main views with only
one view displayed at a time.
S_F6_R5 80 The system must have to be integrated in the
bridge command and control system Parameters: interfaces specification
Capacity to be integrated Taking into account the interfacing constraints
P1
Use of a touch screen for the HMI operation. It should
be easily accommodated on
a ships bridge.
S_F6_R6 81
The system should have to be compatible with existing system (ECDIS, communication channels,
on board wire or wireless network) Parameters: interfaces specification, compliance
with IMO regulations.
Capacity to be integrated Taking into account the interfacing constraints
P1
The HMI receives the vessel sensor data via Protobuf. Another module is
responsible for sending this data
to the HMI.
S_F6_R7 81 The operator should be able to access processed or linked raw sensor data or current or past data.
Ergonomic design
P1
The operator can see both the raw data (AIS, radar, etc.), as well as processed data (Merged Tracks) on the map view. Past data can be
seen in the form of suspicious event on the Timeline
view.
IPATCH D7.32
35
S_F6_R8 81
The system should be able to generate automatic report on pirate attack and send it automatically or
on captain validation The minimum service will be to provide the captain,
pre-formatted report forms Parameters: specification of attack report
Presence of the functionality
P2
The HMI has the data required to
generate a report but at the moment
it is not done automatically. The crew can see the
data from the various views of the HMI and fill a report manually. Main reason for
that is time constraints.
S_F6_R9 81
Interaction MUST be configurable to suit each individual ship’s/crew’s operating procedures and
personnel roles (including on-board security teams)
Interaction MUST be appropriate for different levels of security skill (on-board security guard/advisor,
captain, officer, general crew member…)
Parameterize accesses to
data, levels of intervention, and
levels of automation.
P1
The HMI is a prototype and implementing
different roles does not add much value to the
demonstration at the end of the project. This is therefore out of scope for the
current development.
IPATCH D7.32
36
S_F6_R10 81 Interaction MUST be useable by different
nationalities and literacy levels Multi language
HMI P1
The HMI is a prototype and implementing
different languages does not add much
value to the demonstration at
the end of the project. This is therefore out of scope for the
current development.
S_F6_R11 81
System SHOULD be integrated with the bridge systems so as not to impose greater burden on the bridge crew (i.e. not another new system to watch).
System SHOULD be in as few screens as possible, and as close together as possible
(‘cockpit’ model)
Compliance with these constraints
P1
Movable touch screen. Only one screen is required
for the HMI.
S_F6_Dem_R1 82
The IPATCH demonstration will illustrate the interactions between automatic processing function and operators: generation of alerts, presentation of situation awareness in typical scenarios selected to
illustrate the man machine interface
Demonstration of the mechanisms and acceptance
by a captain
P1 This will be
validated in the demonstrations
IPATCH D7.32
37
S_F7 Allow data recording,
storage and play back
S_F7_R1 83 The system should record raw and processed data Parameters: number of sensors , volume of data, duration of the scenario or the recording window
Duration of the recording
window and associated
volume of data is a sizing
parameter for the design of the
storage device
P1
The HMI stores data for an amount
of time, i.e. until they become
irrelevant to the current situation.
The only case that stores data for the whole lifetime of its operations is about suspicious events on the Timeline
view.
S_F7_R3 83 The system should enable selective accesses to
stored data and replay Delay for access P1
Access to data can be done via the
drill-down capabilities of the HMI in the map view. The replay function has not
been fully implemented in
order to prioritise other functions.
S_F7_R4 83 The system should protect stored data, when
including personal information
Controlled access to
personal data P1
The HMI does not store any personal
information.
IPATCH D7.32
38
S_F8 Enable configuration of the surveillance
system
S_F8_R1 85 The system should be configurable through MMI by specific operators: administrator and analyst
Configuration windows in HMI
P1
The system is fully configurable. However, the configuration
through the HMI does not add much
value to the demonstration at
the end of the project. This has
therefore not been included in the
current prototype.
S_F8_R2 85 The system should operate with several
configuration of sensors, and should allow update of threat data base
Configurations and deployment of sensors will be
defined in the IPATCH study
(WP6 and WP7). Constraints from
the ship configuration
would have to be accounted.
P1 As above
S_F8_R3 85
The configuration HMI should allow sensor parameterisation, optimisation of processing
parameter, the update of threat description data base. The operator should be the analyst.
The configuration of the global system including users access authorisation, network configuration should be the responsibility of the administrator The access to the system should be protected
Analyst HMI Administrator
HMI Management of
users identification and
user profiles
P1 As above
IPATCH D7.32
39
S_F8_Dem_R1 86 The configuration concept should be demonstrated HMI
Configuration windows
P2 As above
S_F8_Dem_R2 86 The sensors configuration Examples of
sensors configurations
P1 As above
S_F8_Dem_R3 86 The optimisation of algorithms parameters. The
update of the threat database.
The logic and the effect of
algorithms parameters should be explained.
The way threat database could
be updated
P1 As above
S_F9 Manage information
exchange with external systems
S_F9_R3 87 The system should generate alerts and information messages (data format and protocol to be defined)
Ability to send alerts and information message
P1
The HMI generated alerts
that are being shown via the
Alarm notifications but their no ability to send those to the ships around
our vessel. This is because he have
no means of testing the
functionality since we have only one
vessel available on the trails.
IPATCH D7.32
40
S_F9_R4 87 System SHOULD take in data from public sources, commercial intelligence providers, military sources
etc.
Various format and protocols
P2
Not implemented. Not in the scope
for concept prototype.
S_F9_R5 87 System COULD provide data to military, flag states, ports, countries involved, and trade
associations
Various format and protocols,
specific analysis of which data
would expected for each
cooperation
P2
Not implemented. Not in the scope
for concept prototype.
S_F9_Dem_R1 87 Alerts emission and transmission would be
evaluated in simulation Demonstration P1 -
D_F1 Provide the captain
information to evaluate piracy
risk
D_F1_R1 90
The tool should propose a risk assessment model taking into account main factors of a scenario. This
risk assessment service should be used at the different step with several level of information. Parameters: Data on piracy a priori available in
data base, observed data quality from S_F2,S_F3, S_F4
Existing functionality.
Relevance of the risk assessment
model
P1
The Recommendations
area and the Decision Support
view provide information on how
to counteract pirate attacks. In addition, the HMI
provides an Electronic
Countermeasures Manual.
IPATCH D7.32
41
D_F1_R2 91
Before the voyage, the captain should be able to connect to piracy web sites, to consult data base, to collect information on piracy courses of action, on current piracy activity on the area, to consult
information on recommended routes and planned convoys
Existing functionality
Easy access to information
P1
The main objective of the IPATCH
project is situation awareness and
early warning. The route planning aspect has not
been implemented, as it has been
covered in other projects (e.g. PROMERC)
D_F1_R3 91 The tool should allow definition of a route:
successive passage points or area or the up loading of a stored route
Existing functionality
P1 As above
D_F1_Dem_R1 91 The functionalities will be demonstrated through
the HMI described in D_F5
HMI demonstration:
access and display of
information
P1 -
D_F2 Provide the captain information
recommendation on use of
countermeasures
D_F2_R1 92 System should allow the operator to consult the
countermeasures manual Presence of the
functionality P1
The HMI provides an Electronic
Countermeasures Manual.
IPATCH D7.32
42
D_F2_R2 93
System should assess the likely effect of a sequence of counter measure face to a typical scenario according to the rules provided in the
manual
Presence of the functionality
Relevance of the recommended
sequences face to scenario
Estimated global effect
P1
The Recommendations
area and the Decision Support
view provide information on how
to counteract pirate attacks. The
actually computation of the recommendations happens on the
Decision Support Tool - D7.2.
D_F2_R3 93
System should propose the list of the recommended countermeasures or next
recommended countermeasure to be implemented during an on-going attack
The list should be up dated depending on the current selection (by the operator) and on the
observed effect on the threat.
Relevance of the selection,
elementary and global effects
P2
The Recommendations
area and the Decision Support
view provide information on how
to counteract pirate attacks. In addition, the HMI
provides an Electronic
Countermeasures Manual.
D_F2_Dem_R1 93 A captain should test the countermeasures
browser Acceptance P1 -
D_F2_Dem_R2 93 Testing of the countermeasures rules face to
typical scenario Estimated effect P1 -
Testing of the recommended sequence on some
selected real scenario
Relevance of the recommended
countermeasures P1 -
IPATCH D7.32
43
D_F3 Record and store data, generate new
incident or scenario model
D_F3_R1 94 The system should record data concerning risk
assessment and countermeasures ( recommended and really implemented)
Duration of the scenario, volume
of the data P1
The HMI does not store such data.
The Decision Support Tool –
D7.2,is considered with the risk
assessment and the
countermeasures recommendations. However, the HMI
includes the Electronic
Countermeasures Manual.
D_F4 Manage information sharing with
external systems
D_F4_R3 96 The system should generate messages (data
format and protocol to be defined) Ability to manage
messages P1
HMI can receive and parse
messages from the IP and Protobuf
channels.
D_F4_R4 System SHOULD take in data from public sources, commercial intelligence providers, military sources
etc.
Various format and protocols
P2
The data used by the system comes
from these sources, but the functionality to automatically import data
remotely has been left for future work.
IPATCH D7.32
44
D_F4_R5 System COULD provide data to military, flag states, ports, countries involved, and trade
associations
Various format and protocols,
specific analysis of which data
would expected for each
cooperation
P2 Not implemented – part of the future
work
D_F5 Allow interaction with human operator
D_F5_R1 99
The HMI should be physically installed in the control room on the navigation bridge and in the citadel, connected to the vessel network that will
support collection of the data from on board sensors and links with external data
Capacity to be integrated
P1
The HMI can run on a PC and only requires a touch
screen and a connection to the
rest of the system. It can therefore
easily be installed on the bridge or in
a citadel.
The HMI implementation will try optimise the work
load of the operator Acceptance by
the operator
To be validated by the stakeholders
D_F5_R10 99
Interaction MUST be configurable to suit each individual ship’s/crew’s operating procedures and
personnel roles (including on-board security teams)
Interaction MUST be appropriate for different levels of security skill (on-board security guard/advisor,
captain, officer, general crew member…)
Parameterize accesses to
data, levels of intervention, and
levels of automation.
P1
This is a requirement for a
commercial product but not a
priority for the demonstration
system.
IPATCH D7.32
45
D_F5_R11 99 Interaction MUST be useable by different
nationalities and literacy levels Multi language
HMI P1
This is a requirement for a
commercial product but not a
priority for the demonstration
system.
D_F5_R12 99
System SHOULD be integrated with the bridge systems so as not to impose greater burden on the bridge crew (i.e. not another new system to watch)
System SHOULD be in as few screens as possible, and as close together as possible
(‘cockpit’ model)
Compliance with these constraints
P1
Use of a minimalistic design on the HMI. Small number of main views with only
one view displayed at a time.
D_F5_Dem_R2 100 The prototypes of HMI will demonstration the browsing mechanisms’ in the different modes
Feasibility of the decision tool
P1 -
D_F6 Enable the configuration of
the Decision support software
D_F6_R1 100 The system should be configurable through MMI by specific operators: administrator and analyst
Configuration windows in HMI
P1
This is a requirement for a
commercial product but not a
priority for the demonstration system. The
demonstration system is fully
configurable but not directly through
the HMI.
IPATCH D7.32
46
D_F6_R5 101 The access to the system should be protected
Management of users
identification and user profiles
P1
This is a requirement for a
commercial product but not a
priority for the demonstration
system (which is only accessed by members of the consortium so there are no
privacy or security risks).