aarmstrong_project_January2010Page i
DEVELOPMENT OF A METHODOLOGY FOR DERIVING SAFETY METRICS FOR UAV OPERATIONAL SAFETY
PERFORMANCE MEASUREMENT
Andrew J Armstrong
This report is submitted to satisfy the project requirements of the Master of Science in Safety Critical Systems Engineering
at the Department of Computer Science
January 2010
Number of words = 40,088 as indicated by the Microsoft Word ‘word count’ tool. The count includes the title page, preliminaries, report body, and the references.
Page ii
Abstract There is increasing potential for missions utilising Unmanned Air Vehicle Systems (UAS) that will require them to access the same airspace as manned aircraft. Currently the accident rate exhibited by UAS is perceived to be too high and safety improvements are required. A literature review has identified the following operational safety issues:
• the accident history of UAS shows that many contributory causes are due to airworthiness and human factors (HF) issues prevalent in operating UAS;
• operations of manned aircraft domain can be argued as sufficiently safe by
operational safety cases (OSC) which use the Goal Structuring Notation (GSN) to express safety claims, goals and evidence in a structured hierarchical way; and,
• there is limited evidence of suitable metrics for UAS operational safety parameters.
A safety performance methodology has been developed to enable the derivation of safety metrics from fragments of an OSC GSN within the ARP5150 safety assessment framework. The methodology has been tailored to identify suitable performance metrics for UAS and validated by application to a UAS case study. The project has concluded that:
• The direct derivation of safety performance metrics from current OSC GSN fragments is not straightforward.
• A goal based method for deriving metrics can be used in conjunction with GSN safety
cases to more effectively identify metrics.
• Existing GSN constructs can be assessed and annotated with review symbols to identify revisions aimed at improving metrics derivation.
• In a case study it was demonstrated that UAS specific metrics could be identified
using the methodology developed in the project, with the caveat that the military aviation SMS is broadly equivalent to that mandated in civil aviation under ICAO 9859 [Ica06].
Page iii
Statement of Ethics Ethics in Student Projects [Yor08] identifies the basic principles as:
1. Do no harm; 2. Informed consent of human participants in project; 3. Preserve the confidentiality of data held on individuals.
To do no harm to anybody taking part in the project they should not be put in a position of physical danger or asked to do anything which is illegal or against their best interests. In producing the project, literature and case studies have been chosen mainly from extant material in the public domain. Where not in the public domain, permission has been granted for use. Therefore nobody has been put in a position of danger, illegality or been forced to act against their best interests during the production of the project. The project is theoretical and examines GSN argument fragments for the purposes of defining safety performance metrics. As a result of reading this project it is possible that someone may be influenced to modify their approach to the work that they perform in a safety critical industry. It must be stressed that the work in this project reflects the personal views of the author and does not constitute part of any regulation, standard or requirement. The work has been undertaken for the purposes of academic study only. Anyone who reads this project must proceed with caution if attempting to apply any of the findings in practice and they are strongly advised to seek independent opinion from suitably qualified and experienced practitioners before doing so. As there are no active participants in the project, and as it is theoretical, no informed consent is required; no data has been taken regarding active participants and so confidentiality of data is not applicable.
Acknowledgements I would like to thank the following people without whose help and encouragement I would not have been able to complete this project. I thank my parents Andrew and Margaret who have always encouraged me in my studies and passed on their value of lifelong learning. I also thank Richard and Rhiannon for providing practical assistance with a quiet space in which to work. I thank Dave Venn and Phil Barwell of QinetiQ Ltd for supporting my request to move into systems safety, and my colleagues Colin Blagrove, Rod Angel and John Gallacher for their support. I thank Mike Cusack of the Tornado Project Team for permitting the use of the Tornado specific project GSN examples. I also thank Sqn Ldr Kevin Keen of DARS for discussion on Operator top level safety management issues. I thank my project supervisor Dr Mark Nicholson for his advice, guidance and encouragement throughout the development of this project and for the many stimulating discussions we have enjoyed, on systems safety management, during the course Finally I would like to thank my wife, Elizabeth, and my children Timmy, Rhydian and William for their patience, support and giving me the time I required to produce this project.
Page iv
2 Literature Search.................................. ................................... 3 2.1 Unmanned Air Vehicle Systems (UAS) ........................................................................ 3
2.1.1 Definitions of a UAV and UAS ............................................................................. 3 2.1.2 Applications for UAVs .......................................................................................... 5 2.1.3 UAV Benefits and Capability Development ......................................................... 5 2.1.4 Constraints affecting UAV/UAS integration in non- segregated airspace ........... 7 2.1.5 Safety Regulation of UAVs .................................................................................. 7 2.1.6 UAV Hazards and Safety Objectives ................................................................. 11 2.1.7 UAV Accident rates............................................................................................ 12 2.1.8 Autonomy........................................................................................................... 13 2.1.9 Work by York University MSc Students in UAV safety. ..................................... 16 2.1.10 Summary of Sections 2.1.1 – 2.1.9.................................................................... 16
2.2 Operational Safety Management................................................................................ 17 2.2.1 The Origins of the Safety Case.......................................................................... 18 2.2.2 The Requirements for Operational Safety Cases.............................................. 20 2.2.3 OHSAS 18001 ................................................................................................... 22 2.2.4 Alternative OSC structures for UAS................................................................... 23 2.2.5 Accident – Cause Models .................................................................................. 25 2.2.6 Military Aviation Safety Management Systems.................................................. 26 2.2.7 Civil Aviation SMS - ICAO 9859 Safety Management Manual .......................... 27 2.2.8 Civil Aviation SMS in the US and Canada......................................................... 27 2.2.9 Summary of sections 2.2.1 - 2.2.8 ..................................................................... 28
2.3 Safety Performance Measurement............................................................................. 28 2.4 Safety Monitoring........................................................................................................ 30
3 Developing a Methodology for Safety Performance Measurement ........................................ ...................................... 40
3.1 Introduction ................................................................................................................. 40 3.1.1 Step 1 – Consider the components of a GSN fragment .................................... 41 3.1.2 GSN Fragment Searching Process ................................................................... 43 3.1.3 Step 2 – Select generic GSN fragments & define metrics................................. 45 3.1.4 Step 3 Evaluate the results ................................................................................ 53 3.1.5 Step 4 –Fixed wing instantiation of the OSC fragment G2.2 ............................ 55 3.1.6 Step 5 - Evaluation and Methodology Revisions .............................................. 58
3.2 Summary of Methodology........................................................................................... 60 4 Case Studies....................................... ................................... 61
4.1 Case Study 1 – High level Aircraft Operator Safety Argument .................................. 61 4.2 Case Study UAV MQ9- Predator B – “Counter factual” GSN example...................... 68 4.3 Summary .................................................................................................................... 76
5 Project Conclusions................................ .............................. 77 6 Future Work ........................................ ................................... 79 7 References ......................................... .................................... 80
Page v
Figure 1 - Diagram of a UAV showing the main functional interfaces....................................... 4 Figure 2 – Examples of the range of UAVs by weight categories replicated from [Wei04]....... 6 Figure 3- UAV Autonomy levels from HERTI program [Wil09]................................................ 14 Figure 4 – Aircraft system operational safety case modelled on the NATS approach............ 24 Figure 5– ARP 5150 Safety Assessment Process .................................................................. 34 Figure 6– Fragment of Generic GSN....................................................................................... 41 Figure 7 – Example GSN for Systematic GSN review process............................................... 44 Figure 8 –Process for searching nodes in the GSN fragment of interest................................ 45 Figure 9 – Top level goal for the generic OSC. ....................................................................... 46 Figure 10 – OSC Legislation and Regulation Pattern ............................................................. 47 Figure 11 – Generic OSC Risk Management Pattern ............................................................. 48 Figure 12 – AOA safety argument High level GSN ................................................................. 62 Figure 13 - Supporting goal structure to S11 .......................................................................... 64 Figure 14 - AOA safety argument Strategy S12...................................................................... 65 Figure 15 – AOA safety argument Strategy S13 ..................................................................... 66 Figure 16 – AOA safety argument Strategy S14 ..................................................................... 67 Figure 17 –Predator UAV types............................................................................................... 68 Figure 18 - Predator Ground control station............................................................................ 69 Figure 19 - Top level representation of the “counter factual” safety argument ....................... 71 Figure 20 – Unsafe organisational influences “counter factual” GSN ..................................... 72 Figure 21 – “Unsafe supervision issues” counter factual GSN ............................................... 73 Figure 22 - Preconditions leading to unsafe acts GSN counterfactual.................................... 74 Figure 23 - Unsafe Acts GSN counterfactual .......................................................................... 75
Table 1 –Interim classifications of UAS from CAP722 [Caa08] ................................................ 8 Table 2 – Evaluation of the suitability of goal based metrics to a GSN based OSC............... 36 Table 3 – example of GQM approach from [ Bas94]............................................................... 37 Table 4 – Results of metric derivation for Figure 11 OSC risk management. ......................... 50 Table 5 - The GQM method applied to goal G2.2a (operational risks are identified)............. 51 Table 6 - The GQM method applied to goal G2.2b (operational risks are assessed)............. 52 Table 7 - The GQM method applied to goal G2.2c (operational risks are properly managed) 52 Table 8 – Proposed Metrics <Mark-up> for GSN reviews...................................................... 54 Table 9 – Description of <Operator> from [Hrm07]. ................................................................ 54 Table 10 - Tornado “top level” goals for risk management using the direct derivation of metrics method ........................................................................................................................ 57 Table 11 – Analysis of nodes from Tornado OSC below Goal G2.2.1.2................................. 58 Table 12 – Summary of case study requirements................................................................... 61 Table 13 – Metrics derivation for AOA safety argument (high level strategies) ...................... 61 Table 14 - Metrics derivation for AOA safety argument (top goal) .......................................... 63 Table 15– Proposed metrics and analysis for goals under Strategy S11 ............................... 64 Table 16 –Proposed metrics and analysis for goals under Strategy S12 ............................... 65 Table 17 – Proposed metrics and analysis for goals under Strategy S13 ............................. 66 Table 18 - Proposed metrics and analysis for goals under Strategy S14 ............................... 67 Table 19 – Specifications of predator UAV ............................................................................. 68 Table 20 - HFACS analysis from Carrigan [Car08] ................................................................. 70
Page 1
1 Introduction
1.1 Background There is increasing demand and commercial potential, for using Unmanned Air Vehicle Systems (UAS) for airborne missions that will require them to access the same airspace as manned aircraft. Currently the majority of UAS operations are limited to segregated airspace where operations can be conducted without threat of harming other airspace users or third parties on the ground. Access to non-segregated airspace will require UAS to show an equivalent level of safety (ELOS) as manned aircraft and appear transparent to other users and Air Traffic Management (ATM). In support of emerging regulation and the need to operate UAS within a civil safety management system it is apparent that effective safety performance measurement will be an important future enabler in support of the demonstration of adequate safety achievement.
1.2 Motivation and Aims for the project Many civil variants of UAS are, or will be, based on military designs and so re-use of safety evidence to qualify against civil certification requirements can be expected. Military UAS are operated within a safety management system that emphasises a risk management approach to safety. Operational Safety Cases (OSC) are used to argue that the design is sufficiently safe to operate in a given environment for a given task. However, in complying with emerging regulations, UAS will be required to demonstrate adequate levels of safety in accordance with civil standards and the SMS. Based on the above scenario, the aim of the project was to identify and begin to address the requirement for a safety performance measurement method based on an existing OSC to provide information to suit the requirements of civil standards and emerging UAS regulations. It is recognised that OSC arguments are expressed in Goal Structuring Notation (GSN) that links safety objectives (goals) to evidence (solutions) in a hierarchical way. The aim of the project is to derive safety metrics from this pre-existing material as it should enable measurement of the key objectives within the context of the safety case.
1.3 Project Scope and Limitations The scope of the project is to examine fragments of GSN arguments from pre-existing operational safety cases to determine if it is possible to identify suitable metrics to enable safety performance measurement for UAS. A limitation is that the extant material may not be concerned with UAS specifically. It is assumed that this is not sufficiently limiting to prevent a general method being developed as a first step. Once a method has been established, a case study that includes UAS specific safety arguments (expressed in GSN) should be suitable to validate the method. The majority of the OSC fragments are derived from military applications as these are the most commonly found examples of UAS usage. This presents a limitation for civil applications because such OSCs have been developed to fit within a military aviation SMS. In principle it is assumed that the basis of military and civil SMS are close enough that OSC material from the military domain is a suitable basis on which to define metrics for use within a civil SMS. This assumption is based on recognition that the underpinning principles of SMS are based on general systems safety standards that are very similar. In particular the ICAO Safety Management Manual (SMM) 9859; Aerospace Recommended Practice ARP5150 - Safety Assessment of Transport Airplanes in Commercial Service; CAP722 and CAP740 are
Page 2
considered to be the primary applicable standards of interest that define the civil environment for UAS integration in non-segregated airspace relevant for metrics development. The scope of the project includes utilising existing methods to derive metrics from GSN fragments. It is considered out of scope to source information external to the safety case GSN to define a metric, as this would not be faithful to the objectives of the project. Within scope is considered demonstrating inadequacies with a current OSC approach that affect metrics derivation. It is considered to be out of scope to create new OSC constructs or judge whether an OSC is fit for purpose. The scope of the literature review is to: identify the key UAS safety issues that will determine the requirements for safety performance measurement; understand the basis for operational safety cases; and, identify the role of safety performance measurement within the overall SMS. This is needed in order to understand the application and context for metrics development. The methodology proposed by the project will be evaluated by case study. Such case studies will need to be based on a UAS or have applicability to a UAS.
1.4 Report Structure Section 2 covers the literature search which includes the topics of UAVs, Operational Safety, Safety Management Systems and Safety Performance Measurement. This section describes the findings from the literature review and justifies the subsequent approach taken by the project in the design and implementation phases. Section 3 describes the development of the methodology for the derivation of safety metrics by considering the application of the first step of the ARP5150 [Arp03] process to suitable GSN expressions of safety arguments. Section 4 describes the results and evaluation of applying the devised methodology to two different case studies. Case study 1 comprises a top level organisational safety argument from an aircraft operator‘s viewpoint and is based on pre-existing fragments of GSN. Case study 2 is a UAV GSN based on a “counter-factual” argument constructed by the author based on information from published literature. A counter factual argument is an argument demonstrating the absence of safety. It was necessary to do this as GSN safety arguments for UAS that were sufficiently detailed were not available. Section 5 identifies future work based on the findings of the literature search, methodology development and evaluation by case studies.
Page 3
2 Literature Search This section describes the findings from the literature review and justifies the subsequent approach taken by the project in the design and implementation phases.
2.1 Unmanned Air Vehicle Systems (UAS) The purpose of the literature search on Unmanned Air Vehicles (UAV) is to understand the relevant safety management issues that directly affect the feasibility of flights in non- segregated airspace and thus determine which areas of UAV safety performance measurement it would be appropriate to investigate.
2.1.1 Definitions of a UAV and UAS In CAP722 [Caa08] the UK CAA provide advisory definitions for the terms UAV and UAS, and also list definitions used by the military as found in Joint Service Publications (JSP) 550 [MOD06] and JSP 553 [ MOD08]. A UAV is defined as “An aircraft which is designed to operate with no human pilot on board as part of a UAS” [Caa08] section 2.1. Where, “An Unmanned Aircraft System (UAS) comprises individual 'System elements' consisting of the unmanned aerial vehicle (UAV), the Ground Control Station (GCS) and any other UAV System Elements necessary to enable flight, such as a Communication Link and Launch and Recovery Element. There may be multiple UAVs, GCS or Launch and Recovery Elements within a UAS.” CAP722 acknowledges that the GCS may be on board a ship or land based. The European Aviation Safety Agency (EASA) [EAS05] provides a very similar definition but includes a description of the applicable phases of flight as well “taxiing, takeoff and recovery/landing” Military definitions of a UAV are very similar but include the military purposes of a UAV. This is exemplified by JSP 553 [MOD08] which states for a UAV: “A UAV is defined as an aircraft which does not carry personnel and: is capable of sustained flight by aerodynamic means; is remotely piloted or automatically flies a pre-programmed flight profile; is reusable; is not classified as a guided weapon or similar one-shot device designed for the delivery of munitions.” The UK MOD does not use the acronym UAS instead it defines the term UAVS as the Unmanned Air Vehicle System. In this project the acronym UAS will generally be used. The US military definition of a UAV is given in the US DoD Joint Publication 1-02 DoD Dictionary [Usd01] and is very similar to UK definitions with the exception that it allows for expendable use. In order to illustrate the above concepts, a typical architecture for a military UAV is shown in Figure 1 replicated from the US Office of the Secretary of Defense (OSD) Unmanned Aircraft Systems Roadmap [Osd05]. The UAS comprises a vehicle and payloads, a command and control system and communications architecture. Communications are achieved beyond line of sight (BLOS) via a satellite communications (SATCOM) link and by line of sight (LOS) from a control station (CS).
Page 4
Figure 1 - Diagram of a UAV showing the main functional interface s The UAV is a fully functioning, airworthy, flight capable vehicle comprising functions for flight control, payload control, weapons employment and situational awareness. The flight control function is necessary to maintain or change the required flight path and other flight parameters such as velocities, height, and accelerations in response to externally received commands or internally from an autonomous control function. The payload control and product dissemination functions serve to switch on and control onboard sensors. This normally involves still photography or Full Motion Video (FMV) collected from optical or Infra Red (IR) sensors and the subsequent packaging of collected optical or video data for transmission to a ground station. A UAV which has been designed specifically for the military requirements of combat or strike missions is called an Unmanned Combat Aerial Vehicle (UCAV). It should be noted that some commentators make a distinction between a UAV that has a weapon added as a secondary function and a purpose designed strike capable UCAV. For example Predator B was originally designed to be a surveillance UAV and has had weapons added (e.g. hellfire missiles) by modification. This is primarily a UAV and known as such. For development projects a CS can be as straightforward as a laptop linked to a transmitter / receiver for LOS operations conducted on a range. For production systems the CS is likely to be more sophisticated and can comprise UAV pilot (UAV-p) and camera controller stations linked to multiple communication data links. Control may be achieved by the UAV-p and a sensor operator working together to interpret information received from the data-link. Control may be handed over from one UAV-p to another. One UAV-p may control or supervise the operation of the UAV during the mission phase of flight. A different pilot may be employed during the take-off, climb, cruise and approach/landing phases of flight. In such cases, control handovers from one pilot to another are required to maintain continuous flight and for endurance missions handover may be necessary to relieve pilots in accordance with duty rosters. The UAV-p is defined by CAP722 [Caa08] as “the person in direct control of the UAV”. In this role the UAV-p monitors the data provided by the flight control and situational awareness functions and provides control commands back to the UAV to ensure that it follows the required flight path. There is also the term UAS Commander defined that allows for a fleet of UAVs in a UAS being commanded by a single supervisor. The UAV-p is the direct operator of the UAV; however the term operator has another meaning in the context of the organisations that operate the UAS. The term Operator or Air Operator is assumed to be the airline in the civil standard ICAO 9859 Safety Management Manual [Ica06] paragraph 2.1.13. Another civil
Page 5
standard, CAP722 [Caa08], defines the operator as the “legal entity operating a UAV System”. The operator is actually considered to be one stakeholder in the complex system of aviation. In civil aviation “operations” are described as being dependent upon service providers in addition to, and separate from, the operator. Service providers are described in [Ica06] as comprising air traffic management; aerodrome operations, including airport emergency services; airport security; and navigation and communication aids.
2.1.2 Applications for UAVs The US Road Map [Osd05] describes the main applications for UAVs as fulfilling the dull, dirty and dangerous missions as this relieves humans of these categories of missions providing economic, reliability and safety benefits. An example military mission where UAVs can offer benefits is persistent surveillance, conducted behind enemy lines. Such a mission requires extended time on task, in order to capture photographic or video evidence of potential targets or threats, and there is a high risk of attracting hostile enemy action. Clearly this mission can be both fatiguing and dangerous for human beings in manned aircraft. Using a UAV for flights over enemy territory reduces the direct safety risks to aircrew but can introduce a security risk, if the UAV were to be captured by the enemy that may precipitate a hazardous mission to recover or destroy it [Osd05]. There are many potential civil applications for UAVs cited in the literature. For example De Garmo [Deg04] investigated and summarised a number of potential civil applications reported by others (e.g. reported Frost and Sullivan briefings not available in the public domain). Many of these key applications for civil UAVs in the USA derive from the need for enhanced “Homeland Security” provisions following the 9/11 attacks on the world trade centre. Typical security applications include border patrol, monitoring of sensitive sites, drug surveillance and interdiction, domestic traffic surveillance, pipeline patrol and port security. Other roles reported by De Garmo [Deg04] include: emergency response, law enforcement surveillance, search and rescue, forest fire monitoring, flood mapping, nuclear biological chemical (NBC) monitoring and chemical and petroleum spill monitoring. One notable success for UAVs in the civil market is their use in agricultural crop spraying in Japan where it is reported that around 2000 vehicles are used commercially [Osd05].
2.1.3 UAV Benefits and Capability Development UAVs vary considerably in physical size, weight, range, speeds, application and endurance. An illustration of the range of weights and types of contemporary UAVs is shown in Figure 2 replicated from Weibel and Hansman [Wei05]. At one extreme are small hand portable devices, weighing a few ounces and fitted with miniature cameras, that can be used by individual soldiers to observe what is in a building or in the next street beyond their immediate line of sight. At the other extreme are the much larger, heavier and more complex Medium Altitude Long Endurance (MALE) and High Altitude Long Endurance (HALE) UAVs, such as those used for persistent surveillance missions.
Page 6
Figure 2 – Examples of the range of UAVs by weight categories r eplicated from [Wei04] One set of benefits of a UAV is that it will not experience flight control or navigation failure modes due to pilot fatigue, failed life support systems, degraded visual conditions or the presence of smoke in the cockpit. For a UAV that lands or takes off under autonomous control, there is no pilot indecision failure mode possible [Osd05]. However the removal of the pilot may lead to the design organisation reducing the level of system redundancy or using lower quality components than were previously specified for aircrew safety, thereby putting affordability before reliability and airworthiness considerations [Osd03]. A significant contribution to combat aircraft design costs and operating limitations is due to the need to provide a safe environment for the human aircrew that operate the aircraft. Removing the pilot from an aircraft removes the need for life support systems, a cockpit or flight deck and is reported as saving 3000-5000lbs mass from an aircraft [Osd05]. This permits the UAV designer to expand the flight envelope considerably in terms of speeds, accelerations (g- levels) and manoeuvrability giving the prospect of enhanced survivability for certain military missions. [Osd05], [Wez07]. Within the last decade there have been some noteworthy illustrations of UAV developments. In August 2001 the NASA Helios prototype UAV set an altitude record of 96,863 ft. This was a prototype and it remains a challenge to design a practical air vehicle capable of managing to climb to such an altitude, sustaining a cruise and carrying a useful payload [Cox04]. In 1998 the Aerosonde Mark 1 “Laima” became the first UAV to cross the Atlantic Ocean achieving an altitude of 1,680 m, and completing the journey of 3270 km in 26 hours and 40 minutes using just 7 litres of fuel [Bar98]. This clearly demonstrated the potential capabilities of endurance and lower cost of operations when compared with manned aircraft. Perhaps more significantly, in terms of technology readiness and regulatory readiness, on the 18th August 2005 BAE Systems achieved the first CAA approved fully autonomous mission of a UAV in UK airspace [Wil09]. Cox et al [Cox04] provides information and analysis of US civil technology developments. For example the NASA led Autonomous Robust Avionics (AuRA) programme is developing technology to enable aircraft to fly with reduced or no human intervention; to include the capability to optimize flight over multiple regimes, and the ability to provide maintenance on demand. Three main components of AuRA are Intelligent Mission Management (IMM), Integrated Systems Vehicle Management, and Adaptive Flight Controls. Autonomous control capability is also a significant enabler for the military where one of the aims of technology
Page 7
development is to delegate the basic flying of the vehicle to autonomous control and thus enable humans to concentrate on mission decision making tasks [Cox04] In the European Union the most urgent requirements were considered to be for increased numbers and capabilities in MALE and HALE UAVs for their long endurance surveillance capabilities and UCAV for their enhanced performance in dangerous missions [Wez07]. All current European UCAV programmes are reported to be technology demonstrators. For example UK industry is leading the development of Taranis and Corax [Wil09]. Taranis is a demonstrator programme the size of a small combat aircraft powered by a turbofan engine and will have intercontinental range. Corax is essentially a stealth variant of the same UCAV and large enough to be functional. Such developments pose a challenge for designers and operators in considering how such future UAS can be practically catered for within the existing airspace management arrangements. These UCAVs are likely to be flying long range, hard to detect, and operated under autonomous control. Their performance in terms of speeds, accelerations and manoeuvrability is likely to be far in excess of the Typhoon aircraft. Typhoon currently represents the upper limit of segregated airspace protocol classification listed in CAP740 [Caa07b] Chapter 2 Annex A. It is not likely that such a protocol will apply to all of these future UCAVs. For more immediate requirements, the UK has made acquisitions of UAS for Intelligence Surveillance Target Acquisition and Reconnaissance (ISTAR) roles for defence purposes, with the intended theatres of operation including Iraq and Afghanistan [Ukg08]. Key UK UAV programmes for current operations have involved UOR acquisitions of Reaper (formerly Predator B), Hermes 450 and Desert Hawk 3 (DH3). In future scenarios it is predicted that swarms or teams of UAVs could operate cooperatively to achieve a given mission purpose [Clo02b]. While an interesting prospect for the future (probably military missions), there are currently many constraints limiting the use of a single UAV in non-segregated airspace.
2.1.4 Constraints affecting UAV/UAS integration in non- segrega ted airspace To fulfil many of the identified civil missions for UAVs will require them to have routine access to non-segregated airspace. However there currently exist a number of constraints that will need to be overcome. With reference to integration of UAS within the US National Air Space (NAS), De Garmo [Deg04] concluded that resolution of the following key issues is necessary: 1. The need for a consensus on operational concepts, definitions, and classifications of
UAVs 2. The requirement for certification standards and regulations to address UAS operations
and operator qualifications 3. The provision of effective and affordable collision avoidance systems capable of detecting
non-cooperative airborne threats (e.g. aircraft not fitted with transponders) 4. Improvements in the reliability of UAS and operations 5. Provision of a suitable protected frequency spectrum for communications – considered
out of scope for this project. 6. High insurance liability costs – considered out of scope for this project 7. High acquisition and operational costs – considered out of scope for this project In a separate study, Cox et al [Cox04] investigated and summarised a number of capability and technology issues that are required as part of the perceived solution to achieving integration with manned aviation. They identified that in order to achieve a similar level of reliability to the human pilot of a manned aircraft the UAS would need to demonstrate both system reliability (minimised component failures) and an onboard intelligent decision making capability. The issues identified were: autonomous mission management, collision avoidance, intelligent system health monitoring and reliable flight systems.
2.1.5 Safety Regulation of UAVs The CAA Directorate of Airspace Policy has published CAP722 “Unmanned Aircraft System Operations in UK Airspace – Guidance” [Caa08]. This document, now in its third edition,
Page 8
provides advice to the developers of a UAS on how to identify the route to certification and ensure that all the relevant standards and regulations are complied with. It also provides guidance on the safety requirements for airworthiness and operational standards that have to be met if a UAV is to be permitted to fly in UK airspace. Currently in the UK, UAV flights beyond “the limits of visual control” are not permitted outside of segregated airspace. For all UAVs that are permitted to fly, CAP722 states: “It is CAA policy that UAS operating in the UK must meet at least the same safety and operational standards as manned aircraft.” [Caa08]. This is true for all aspects of UAS operations, in the air and on the ground, that they meet the same safety standards as equivalent classes of aircraft. This is known as Equivalent Level of Safety (ELOS). Evans [Eva06] reviewed UAV regulations and he concluded that whereas the majority of UAV regulation is based on the ELOS principle, there was not much guidance provided on how this should be achieved. Therefore a potential difficulty for designers and operators of UAS is how to demonstrate compliance with ELOS such that the regulators will issue permits to fly. This has implications for the measurement of safety performance. In order to issue a permit to fly the authorities will need to be convinced that the safety performance will meet ELOS. However if the contributory processes and the associated measures of success are not fully known how can an operator provide the requisite evidence to justify that operations will be safe? The starting point for ELOS is to determine the correct classification of the subject UAV against an appropriate civil aircraft category. At present there is no internationally agreed classification system for UAS although one recommended way forward is to base such a classification scheme on how the UAS will be operated in civil airspace [Deg04]. The UK CAA [Caa08] acknowledge that the process of designing a classification system is not yet complete and as an interim measure they have provided guidance on a suitable classification scheme as shown in Table 1.
Table 1 –Interim classifications of UAS from CAP722 [Caa0 8] Although the CAA show interim classifications these are not actually related to manned aircraft categories in all cases and so it is not possible to identify the ELOS requirements from Table 1. Classification depends instead on calculation of kinetic energy levels. For instance CAP722 requires UAVs in class 2 (20-150kg mass) to have a kinetic impact energy of less than 95kJ. Guidance published by Haddon and Whittaker [CAA02] considers equivalence from a comparison of kinetic energies between UAVs and equivalent aircraft classes. Their approach recognised that an air vehicle can harm third parties on the ground in proportion to its level of kinetic energy upon impact. There were two impact scenarios considered in calculating levels of kinetic energy. The “unpremeditated descent scenario” was defined as a failure condition that results in the inability of the air vehicle to maintain a safe altitude above the ground. The “loss of control scenario” was defined as a failure condition that results in a loss of control that may lead to a high impact velocity. Haddon and Whitaker derived formulae for calculating kinetic energies for each of these scenarios and produced results and exemplars for different categories of air vehicle (e.g. JAR 23 and JAR25) in terms of kinetic
Page 9
energies. In order to determine equivalence the kinetic energies for both scenarios “unpremeditated descent” and “loss of control” were calculated for the subject UAV and compared with the results for the baseline aircraft categories. The ELOS principle is stated as a set of requirements such that a UAS will appear to act and behave to all other users (and operators) of airspace as though it were a manned aircraft. [Caa08]. This is stated in terms of the harm the UAS can potentially inflict to other airspace users and third parties on the ground; and in terms of UAS behaviour characteristics matching those of a manned aircraft (e.g. in response to threat of collision, response to commands from ATC). However there are significant differences between UAS and manned aircraft that have led to the introduction of appropriate new regulations and the reinforcement and explanation of others so that ELOS requirements are more fully described. The key source of these requirements in the UK is CAP722 [Caa08] which will be relevant for the subsequent investigation of suitable safety performance metrics. Current regulations limit the risks of operating certain classes of UAVs by limiting their flights to within segregated airspace, particularly where a UAV does not comply with the Air Navigation Order (ANO) [Caa09]. However, individual flights outside of a danger area (DA) may be permitted by the use of a Restricted Area (Temporary) RA (T) [Caa08]. This provides exclusive use of the airspace defined by the RA (T), to the UAS operator, and will consequently temporarily deny airspace use to other users. This is not a practical means for achieving routine flights in non-segregated airspace as the use of a RA (T) is actually a form of segregation. Generally the provision of Air Traffic Services (ATS) must be transparent to the Aircraft Controller and s/he should not have to do anything different in respect of communications, rules or procedures from that required for a manned aircraft. Furthermore the UAS is required to comply with any instructions issued by ATS and should be equipped for the class of airspace in which it operates. One notable difference is that the UAS should include the word “UNMANNED” on the first communication with ATS. [Caa08]. A further requirement is to comply with instructions given by ATS in a similar timeframe as an equivalent manned aircraft. This raises the question about what is a suitable timeframe. For a manned aircraft the typical time it takes for the pilot to respond to a time critical call from ATC (e.g. to manoeuvre to avoid a collision) has been measured and analysed in a study by Cardosi and Boole [Car91]. They studied the results of 80 manoeuvre calls occurring in 46 hours of recorded en-route time. They measured the durations for each stage of the communication process. The total communication time varied from 4 to 40 seconds with a 50th percentile median time of 9 seconds. In theory this sets a benchmark for a UAS to demonstrate equivalence. Some of the individual processes within the total time are relevant for determining the equivalence standards for UAS. The duration of the initial call from ATC varied between 1 and 11 seconds with a mean of 4.85 seconds. The lag before a pilot’s first response varied from 1 to 31 seconds with a mean of 3.31 seconds and the pilot’s response time varied between 1 and 11 seconds with a mean of 2.61 seconds. Regulations require a response from the UAS of a “few seconds” to ATC commands [Caa08] and this agrees with the manned pilot measures of the study [Car91]. Some other interesting aspects of the study were to note that 6% of first calls received no pilot response and 14% required a second call by ATC to clarify. In these cases the total communication time took a further 4.98 seconds on average to complete the second ATC call and pilots response. Interestingly there was no statistical significance between high and low workload data i.e. the time to respond was independent of pilot workload. The study provides some interesting results for ATC and pilot commands that could be interpreted as setting standards that UAS have to achieve for equivalence. A UAS and a manned aircraft both may not respond to local ATC commands. Both the UAS and the aircraft will be tracked by ATM radar. However the UAS presents alternative means of communication with the UAV-p as s/he is located on the ground. What is not clear is if a non- voice means of communication between ATM and a UAS could achieve a faster means of time critical calling. The known benchmarks for manned aviation set the minimum standards
Page 10
that the UAS must achieve. For a remote UAV-p the latency of the voice communication link plus latency of the command link to the UAV will be an additional factor in retaining control of the system and dependability measures for communications will be an important area for UAS [Mou01]. For manned aircraft the pilot has a role in mitigating the hazard of an air to air collision by implementing the rules of the air by the “see and avoid” approach. For UAVs to be operated in non-segregated airspace, outside of the LOS of the UAV-p, this is replaced by the “sense and avoid” requirement of CAP722 [Caa08]. These require an equivalent capability to human “see and avoid”, and this need is open to challenge. There is evidence that the pilot “see and avoid” capability is significantly unreliable as a means of preventing air to air collisions. De Garmo [DeG 04] summarised the work of various sources from the literature. A significant point was the reported variation between individual pilots who may spend different times looking out of the window, who may have different visual abilities and follow differing scanning techniques. He reported on work conducted by Lincoln Labs that concluded that pilots were not good at identifying potential collisions if they were not previously aware of aircraft in their vicinity. De Garmo quoted FAA data that showed that many air to air collisions took place in daylight near uncontrolled airports. Based on this evidence, De Garmo challenged the value of basing a sense and avoid requirement on an obviously poor human equivalent. A sense and avoid capability actually has the potential to be much better than a human in terms of field of view, detection reliability, persistence and tolerance to different weather conditions. The key word appears to be potential, because there is widespread acknowledgement that there is, as yet, no feasible sense and avoid system available [Deg04]. In addition to avoiding air to air collisions the UAS must be capable of avoiding collision with the ground for obvious reasons of affording protection to third parties. Within Chapter 2 of CAP722 the capability to “detect terrain and other obstacles” is included within the general policy on sense and avoid. Other aspects included are the avoidance of hazardous weather, and performing the functions of safe separation from other aircraft. There is little further guidance on how to achieve a compliant sense and avoid system and the CAA consider it is the role of industry to conduct the necessary research and development of suitable systems for employment in UAS. For operations in non-segregated airspace there is a requirement for Standard Operating Procedures (SOP) for take-off and landing procedures; en-route procedures; loss of control data link; and, abort procedures following critical system failure. These are designed to provide for protection against infringing other airspace users and for protecting third parties. These seem sensible and appropriate and are similar to current practice for manned aircraft. Within segregated airspace, such as UK Danger zones, the UAV has exclusive use and thus there are greater freedoms such as enabling the controlled flights of prototype UAVs that may be substandard in terms of airworthiness. However requirements may be enforced such as: communicating with ATS, and procedures for emergency recovery, loss of control link and for the avoidance of infringing aircraft. Within a danger area, such as a UK range, it is possible to exclude the presence of third parties in the air and on the ground and by range control procedures for staff operating the UAS and the range. As there are no third parties present on the ground a suitable independent flight termination system (FTS) can be employed. For example in the event of certain failure modes of the UAV the FTS could cut-off the fuel supply to the engine, trim the flight control surfaces and return the UAV to the ground in a controlled descent by deploying a parachute [Wei05]. CAP722 does not cover model aircraft, which are provided for by CAP658 [Caa07a]. Some UAVs are small enough to qualify as model aircraft and are flown as such. These UAVs are out of scope for this project. CAP722 does not cover UAS used for the purposes of delivering a weapon and UCAVs and UAVs with weapons capability are outside the scope of this project.
Page 11
Summary of safety regulation issues in respect of safety monitoring The CAP722 requirements discussed above appear to present operators and developers with significant challenges for operating UAS in non-segregated airspace and will influence choices made about such factors as: the concept of operations; the location of the UAV-p; the methods of communication employed; the levels of autonomy in controlling the UAS; the standard operating procedures; the type and extent of operator and UAV-p training; the integrity of systems; and, the size and payload capability of the UAV to ensure it can carry mandated special equipment. Consequently compelling safety arguments and supporting evidence will be required in the operational safety case to satisfy the above aspects together with the identified CAP722 requirements. Additionally the safety management system will need to include an appropriate safety monitoring regime to enable the system to be operated safely.
2.1.6 UAV Hazards and Safety Objectives A key objective of aircraft safety management is to achieve a target level of airworthiness in the original design and preserve a satisfactory or improved level of airworthiness through-life during the operation of the aircraft [Omm08]. This is true for manned and unmanned aircraft in both civil and military sectors. UAS activity presents a hazard to people on other aircraft (from air to air collisions) and to people on the ground. Ground hazard The ground hazard has been analysed by the application of various ground impact fatality models by Weibel and Hansman [Wei05] and Clothier and Walker [Clo06]. The approach reported by Weibel and Hansman uses event tree analysis and a fatality expectation model to determine the level of reliability in the UAS necessary to achieve the required target level of safety. The class of UAV and the area over which it is to operate determines the required level of system reliability. A conservative approach is to use a target probability of a fatality rate of 1 x 10-07 per flight hour. From their calculations the key drivers, limiting where UAVs can be safely flown, are UAV mass and population density. Micro UAVs can be safely flown in the majority of US airspace. MALE UAVs are an “intermediate risk” and should be able to fly over the majority of US airspace but not highly populated areas. HALE UAVs require a reliability rate of no more than one class A mishap per 100,000 hrs (equivalent to military aviation safety levels) to fly over approximately 20% of the USA. Recognising that agreement on safety targets and ELOS requirements for UAS were yet to be harmonised Clothier and Walker [Clo06] investigated and discussed the issues around defining appropriate safety objectives for UAS aimed at meeting the ELOS requirements in emerging regulations. They investigated this by reviewing the actual occurrence of ground fatalities from a survey of National Transportation Safety Board (NTSB) data and challenging what is meant by equivalence. They constructed a simple model to show where a UAV with a given mishap rate per flying hour could crash without exceeding the safety target in terms of ground fatalities per hour. Their analysis of the NTSB accident database [Nts05], over the period 1984 to 2004, revealed that less than 1% of the 27,404 fatalities recorded in the NTSB database were to people on the ground. The vast majority of recorded fatalities all occurred to people onboard aircraft. Furthermore they found that observed involuntary ground fatalities were 3.6 x 10 -08 per flight hour. based on the FAA 3 year rolling average data. Using a target level of safety of 1 x 10 -06 fatalities per flight hour they demonstrated that, for an urban surveillance mission, using Predator B data (mishap rate 170 X 10-06 mishaps per hour) it was not possible to justify operations over, or within 20km of, Brisbane central business district under any feasible safety objectives examined. Mid-air collision hazard The calculations to determine the risk of a mid air collision are complex because air traffic is not uniformly distributed being much denser above large cities with major airports and in specific air corridors. Analysis, by Weibel and Hansman, on the probability of mid air
Page 12
collisions involving UAVs in the US National Air Space, concluded that the ambient collision risk (without mitigation) was of the order of 1x10-07 per hour in areas away from airways and major flight levels [Wei05]. Thus they suggest that small UAVs could be operated in these regions. Flying above scheduled air traffic (at altitude > 50,000ft) is a low risk region which it is claimed could be suitable for HALE UAVs on the assumption that suitable procedures could be developed to enable the aircraft to ascend and descend through the lower altitudes. In other regions additional collision avoidance mitigation strategies will be required. Between flight levels mitigation could be achieved by ATM separation or a capability in the UAV to avoid other traffic. At altitudes below 5000ft the traffic density is such that ground based radar or line of sight collision prevention may be required. Summary of Safety objectives and hazards in respect of safety monitoring The safety targets that should be set for UAV catastrophic accident rates are not clear or consistent. In the literature assumptions are made as to what the target should be and reliability data is used to calculate where a UAV can be flown. In order to produce a reliable measure a consistent benchmark will be required. Until a clear target is set it will not be possible to accurately specify the required reliability of UAVs and this risks over, or under estimating the required safety performance for operations.
2.1.7 UAV Accident rates A presentation by Allouche at the UAV – NET conference in Stockholm 2003 [All01] included qualitative claims for the historical, current and future predicted airworthiness safety performance of UAVs against the general aviation safety level, for the Israeli Industries product range. Four points of note emerged. First: a combination of design features are required to drive accident rates down to equivalent levels of safety to general aviation safety levels. In 1988 the Pioneer design, fitted with single control analogue flight controls and a customised engine had significantly lower safety levels than manned aviation. The Searcher design in 1992 adopted a dual channel, digital flight control system and redundant communications resulting in improved safety. In later designs, adding redundancy to safety critical systems, automating take off and landing and using FAR 33 compliant engines were all improvements that resulted in a UAV design matching general aviation safety levels. Second: the safety gains made followed an exponential curve with a law of diminishing returns applying. Thirdly: the exponential reduction in accident rates appears to mimic the published data for manned aircraft [Wei05] that occurred during the development of the airline industry. Fourth: It is not clear what the context for UAV operations was and what the mishaps were (ground or mid air collisions). Therefore while the reduction in safety is acknowledged as progress this does not imply that the level of safety now achieved will be sufficient to permit flights in non-segregated airspace over a metropolitan area. There is other evidence in the literature that accident rates for UAVs, in the military, are much higher than the picture Allouche presents for civil UAVs. The mishap rates for US UAS are reported to be 10 to 100 times that of manned aircraft [Man04] and Class A mishap rates of 32 to 334 per 100,000 flight hours are reported by the US Department of Defense [Osd03]. However trend analysis published by Weibel and Hansman [Wei05] showed that UAV accident rates since 1987 have been falling and, by extrapolation, are predicted to be able to match the levels associated with general aviation. In summary for the larger UAVs the mishap rate is considered to be too high and this requires improvement before flights in non- segregated airspace can be fully justified. Following analysis of UAS accidents [Wil04] it was concluded that electrical and mechanical reliability of the UAS were as significant as human errors in the causes of accidents. This was attributed in part to lower costs of design and production affecting the component reliability, and system redundancy. The more expensive and better engineered the air vehicle the more likely it was to be reliable. It is not appropriate to generalise because there are different factors and circumstances for each class of UAV [Osd03]. For example in the case of the smaller UAVs some of the aircraft failures were attributed to icing due to the aerodynamic
Page 13
properties of smaller sized aerofoils being affected by the occurrence of a thin layer of ice. Furthermore the effects of precipitation have a more damaging effect in eroding the leading edges of thinner aerofoil sections and in penetrating less effectively sealed compartments. Another trend is the high accident rate observed for UAVs fitted with wooden propellers, as these are highly susceptible to rain erosion [Osd03]. Another observation from UAV accident analysis is the effect of Reynolds numbers [Sto51] [Rey83] and [Osd03]. Aircraft that fly with a high Reynolds number tend to crash less often than those with low Reynolds number. Airliners tend to be larger than UAVs, have higher Reynolds numbers and are less likely to crash from the associated aerodynamic effects than UAVs. While this observation is straightforward to make, it is acknowledged that the detail aerodynamic properties of the low Reynolds number designs and how this affects flying qualities are less well understood and this inhibits the development of suitable design mitigations [Osd03] From the accidents that Williams studied, a compilation of the reported human factors issues showed occurrences in the following categories of: alerts and alarms, display design, take-off error, landing error, procedural error, aircrew co-ordination, weather, and pilot in command [Wil04]. These are all significant from an operational safety perspective requiring: suitable design of the UAV and GCS; the introduction of suitable procedures and training for operators and UAV-p; and, appropriate treatment in the safety case. Suitable metrics for UAS could aim to measure and relate parameters from these categories of HF issues to the measurement of overall safety performance. In order to address the identified HF issues one approach is to investigate the feasibility for increasing the levels of automation involved in flying the UAV. For example the US military are investing in research to automate the take off and landing phases of flight to address the high occurrence of accidents [Osd05].
2.1.8 Autonomy Employing a high level of autonomy in mission planning and flying the UAV enables the manpower and facilities associated with the GCS to be minimised and allows for a UAV commander to control several UAVs [Cox04]. This has to be offset against the requirement to respond to ATM in short timescales and to deal with any emerging hazards of operation. Removing the pilot from the air vehicle means that control of the UAV has to be achieved by either: complete automation; complete direct control from a remote pilot; or, a combination of human and automated control. The correct specification of autonomous flight control, navigation and payload management can serve to reduce safety risks. For example in UAV accidents, a large proportion of the human error causes are reported to occur during the take off and landing phases of flight [Osd03],[Wil04]. Automating these stages should produce consistent behaviour from the UAV as it could be programmed to remain on the ground (if criteria for take off are not met) or to loiter or recover to a safe landing site (if approach/ landing/weather criteria are not met). Autonomy offers the benefits of consistent behaviour in response to events such as collision avoidance where a consistent application of rules required by regulations could be implemented. In theory a UAV, with a sense and avoid capability, preparing for take off could see the approach of other aircraft within a 360 degree field of view and in this regard perform better than a human pilot [Deg04]. Automation can relieve humans of the dull flight management tasks thereby giving more time for mission tactical and strategic level decision making [Osd05]. However increasing levels of autonomy introduce new issues that may impact upon safety elsewhere. For example a study of accident data for Global Hawk, a UAV with relatively higher levels of autonomy has found errors in mission planning. It is noted that mission planning is much more complex for an autonomous UAV and can take many days (up to 237 days is reported) to accomplish. Errors occurred because system operators did not properly monitor the mission planning software and thus could not detect or respond to system errors occurring in operations [Wil04]. In order to achieve the required levels of safety
Page 14
in operations with autonomous control there is a need for higher integrity software and mission planning. In order to characterise Autonomy, so called “Autonomy levels” have been presented in various taxonomies such as Sheridan’s 10 level model [Par00] and Clough’s Autonomous Control Levels (ACL) [Clo02a]. The ACL has 10 levels from the lowest, level 0 – remotely piloted vehicle; to the highest, level 10 - human like, that are measured against the parameters of: perception/situational awareness, analysis/decision making and communication/cooperation. In the ACL, level 7 - “real time multi-vehicle cooperation” seems to meet with the requirements of CAP722 for ELOS. At level 7 the situational awareness is described as the detection of other air vehicles in local airspace with multi-threat detection and analysis capability on board. The decision making is described as being able to compensate for anticipated system malfunctions and hazardous weather, to be capable of evaluating and re-planning the flight path to avoid threats and complete the mission. The communication/cooperation capability is described as collision avoidance, use of third party data for de-confliction and hierarchical cooperation with other air vehicles. The ACL taxonomy describes the required or actual behaviour of the UAV but does not appear to describe what the human does in cooperation with the UAV. Performance metrics will be required not only for the technical system but also for the human working in cooperation with the technical system elements. The HERTI UAS illustrates the role of humans in the concept of the taxonomy of autonomy levels. Mark Kane [Kan07] has presented a taxonomy of levels of autonomy (based on the US Navy Office of Naval Research and as used by SEAS DTC) used in the context of BAE Systems research programmes aimed at demonstrating full autonomy (air vehicle and sensor/imagery). A diagram of this taxonomy is replicated below to illustrate what each level of autonomy means in terms of UAV behaviour and what the role of the human “”commander” or UAV-p is for each level of autonomy.
Figure 3- UAV Autonomy levels from HERTI program [Wil09] HERTI is reported [Mor08] as successfully demonstrating autonomous operation by entry to a pre-defined “Search Area”, and the automatic generation of navigation routes to enable the sensors to search and record images of the required area. It is claimed that the system has also demonstrated automatic target detection and downloaded images to the GCS. The HERTI autonomy scale description of the highest capability level for full autonomy does not
Page 15
appear to match the higher levels of capability for multiple UAVs that are represented in Clough’s ACL [Clo02a]. Therefore appropriate choice of autonomy classification is an important aspect of autonomy measurement. Different phases of flight may involve different levels of autonomous control. The management of the associated hazards of flight is shared by humans working in cooperation with an autonomous system that is designed to perform reliably and take the right decisions consistently within the bounds of the prescribed rules. The question of monitoring arises though. How much trust is placed in the system to perform reliably and exhibit safe behaviour? How is the system going to be monitored to provide information to human operators so that they know they do not need to intervene? Is the human operator going to be able to judge if the system has taken the right course of action and is the human operator capable of intervening? [Lev95]. Clearly all these questions will need addressing if a system is going to be permitted to fly. Many of these questions have been successfully addressed in civil aviation for many years where much of the flight is conducted by computers implemented in auto-pilot systems. During flight, aircrew fulfil the roles of monitoring and supervising the systems and the flight path; communicating with ATM and intervening in emergency conditions. Generally it is the take off and landing phases of flight that are directly flown by the pilot. A key point from a safety perspective is that for a manned aircraft it is still “preferable” for the pilot to complete the take off and landing phases of flight. For a UAV it appears “preferable” for these stages to be automated. Autonomous operation may introduce new errors in supervising, monitoring and intervening in the autonomous operation of the UAV. Even in a fully autonomous situation humans will still be required to intervene in emergency situations [Caa08] and act with a suitable level of proficiency (e.g. by exhibiting effective air vehicle handling skills). The human will require alertness and need to respond in a suitable time frame “of a few seconds” as required by CAP722 [Caa08]. This could limit the distance that a UAV-p can be located from the geophysical location of the UAS. For a long range UAS, handover between UAV-p will be required to maintain the “few seconds” of response time. In addition to the attitude and alertness of the pilot there are other human factors issues such as those relating to the design of the ground station “cockpit”. The UAV-p does not have the same human sensory inputs, about how the aircraft is handling, as the pilot of a manned aircraft would. For a UAV this information has to be provided from machine gathered data. Such data requires effective interpretation to be of value to the UAV-p and s/he is likely to have reduced situational awareness. A contributing factor to poor situational awareness is the delay in commands and information between the operator and the UAV system. Such signal transmission delays can be one second or more and can introduce “temporal and spatial uncertainties” for the operator [Mou01]. Wickens has stated that delays of one second or more can lead to significant errors that can result in total loss of control of the vehicle [Wic92]. Metrics that address system transmission performance (e.g. latency, fidelity, security) will be relevant for safety of UAS. A key set of safety metrics will be concerned with measuring the dependability of communications including the elements of human to human and human to communication system. A pilot of a manned aircraft can respond rapidly to changes in weather, aircraft handling and mission environment and make decisions based on recall of relevant data from complex previously memorised information. The pilot has learned from experience and perhaps UAV control software that learns is required. The results of learning could be replicated across the fleet whereas for human pilots the learning and application is different between each individual. The full implications of the effects on human performance interacting within the UAS are not fully understood or properly managed, as accident reports show. Partly this could be due to the “static” allocation of functions between humans and machines during the design process. It has been noted that whereas machines have been designed to surpass human performance, there is not the same evidence that current systems have been designed to allow the human to surpass machine performance [Han96]. This is a complex area that will require the development of suitable measures to enable designs to be assessed for their impact on human performance and the associated contribution to safety performance of the UAS.
Page 16
2.1.9 Work by York University MSc Students in UAV safety. Following a broad survey of the safety issues present in the UAS domain Andy Evans [Eva06] addressed a specific hazard identification process for UAVs based on the safety assessment approach of ARP4761. The results of the hazard analysis could identify suitable areas for safety monitoring regimes. Chris Hodson [Hod08] investigated the handover procedures between control stations for UAVs He concluded that taking the pilot out of the aircraft raises new issues to be managed including sensory deprivation, dependency on the data link, latency of the data link and the effect this has on the ability to control the flight path of the UAV. Some of these issues were discussed above.
2.1.10 Summary of Sections 2.1.1 – 2.1.9 A review of the literature regarding UAS has established that progress will be required in many areas (regulation, technology, airworthiness and operational arrangements) for UAS to be considered sufficiently safe for routine flights in non-segregated airspace. Therefore, there will be many aspects of the UAS that will require measurement to demonstrate adequate levels of safety. In some cases a clear standard does not exist. For example while it is concluded that ELOS with manned aircraft is required, there is not yet agreement on how this translates into safety targets or measures of performance across all aspects of UAS design and operation. However, in principle it should be possible to derive metrics for relevant safety related parameters as these can be identified from the issues reported in the literature review. A key finding is that the accident rate for UAS is too high due in part to airworthiness, and reliability of systems and operations. These causes will need to be analysed further and suitable metrics derived to measure improvements. Some of these metrics may already exist or could be adaptations of pre-existing manned aircraft metrics. For example the airworthiness and reliability of conventional systems and air vehicle structure could be measured and improvements identified in common with approaches used for manned aircraft. This is outside of the scope of the aims of the project which will are focussed on UAS specific issues. The affect on system elements by removing the pilot from the aircraft cockpit and relocating him/her remotely has wide ranging ramifications for system safety. Other significant factors are: the longer time on task a UAS can achieve and the need to achieve ELOS and appear transparent to other airspace users. These factors combine to introduce novel system requirements, operating arrangements and applications of technology that introduce new safety management issues and hence new safety metrics in order to track performance. Section 2.1 .1 highlighted the role of the UAV commander in controlling several UAVs by individual UAV-p. It also identified the need to handover control from one UAV-p to another. There will be a need to define handover procedures and suitable metrics demonstrating the safety of such procedures within the system. Section 2.1.3 highlighted the need for a consistent classification system for UAS and that metrics appropriate to kinetic energy levels were available to identify equivalence to manned aircraft classes. Future scenarios may require safety monitoring of teams or swarms of UAS operating under autonomous control. This is someway into the future and out of scope of the current project. Section 2.1.4 and 2.1.5 identified the requirements of regulations. These included the emerging methods for determining equivalence to manned aircraft classes. The ELOS principle is broader than this and metrics will be required for autonomous operations, collision avoidance, demonstrating transparency to other airspace users (e.g. communication effectiveness) and the effectiveness and readiness of emergency systems (e.g. FTS).
Page 17
Section 2.1.6 reported on the suitability of safety targets and safety objectives in the context of current UAS reliability and concluded that there is not yet agreement on the appropriate safety target that should apply to UAS. Section 2.1.7 identified the key findings from accident data reported in the literature and the significance of HF contributory causes in accidents and the need for metrics to assist in improving safety performance. Increasing the level of autonomy in UAV flight control, particularly in take-off and landing phases of flight, is seen as a potential solution. Section 2.1.8 investigated autonomy and identified that while levels of autonomy classification exist for the technical system, there would need to be further development to include the human working in co-operation with the technical system and suitable metrics to track performance and identify improvements to design and procedures. It was noted that a UAV-p typically will not have sufficient levels of situational awareness at all times and can thus fall victim to latency in communications links between the UAV and the GCS. Metrics will need to be derived to measure the system communications performance and the level of situational awareness the UAV-p has. Many of the issues identified in sections 2.1.1 - 2.1.8 have implications for operational safety and UAS specific metrics will in many cases be those that measure aspects of operational safety. This is examined further in section 2.2.
2.2 Operational Safety Management In this section of the literature survey key concepts in operational safety were examined with regard to how these relate to the operational safety of UAS. It can be asserted that the majority of accidents actually occur during system operation and therefore “operational safety is concerned with protecting people at risk from harm during the operation of a complex system”. [Omm08] Conducting safe operations with UAVs is of fundamental importance and of equal significance to the inherent airworthiness of the UAV design and the “equipment safety” properties of all the elements of the UAS. This was shown by the accident causes discussed in section 2.1.7. But how is operational safety going to be managed effectively and what are the objectives of operational safety management? In order to understand and assess this there are lessons that can be learnt from how operational safety is managed in other sectors of the economy: oil and gas exploration, railway operations, nuclear power generation; and, how military and civil aviation operations are managed for manned aircraft which has more actual domain affinity with UAS. In the military, JSP553 [MOD08] describes “the four pillars” of airworthiness as: the safety management system, compliance with recognised standards, competence (of people and organisations) and independent assessment. Furthermore airworthiness must be managed in- service i.e. during the operational life of the aircraft. One of the key standards to be complied with is Def Standard 00-56 [Mod07b] which mandates the provision of an equipment safety case that is subject to independent assessment. The safety case must be maintained throughout the aircraft service life as changes are introduced into the design, the equipment operation and conditions of use. UK MOD policy on military UAVS is that they are to be treated as UK military aircraft and are subject to the same regulations contained in JSP 550 - Military Aviation Policy, Regulations and Directives [Mod06]. These place requirements on the Aircraft Operating Authority (AOA) to ensure they have provided sufficient standard operating procedures promulgated as flying orders. The orders applicable to UAVs are described in CAP722 [Caa08] and include “detailing the training, competency, currency, medical requirements and crew duty considerations for all personnel involved in the operation of UAVs.” These are all information requirements for safety metrics.
Page 18
2.2.1 The Origins of the Safety Case The Windscale accident remains the UK's worst nuclear accident to date and the findings of the Windscale Accident Inquiry 1957 [Uka57] paved the way for key safety focussed nuclear legislation enacted in The Nuclear Installations Act 1959 [Ukg59]. This established the Nuclear Installations Inspectorate (NII) to act as an independent regulator of the nuclear industry and became law under the Nuclear Installations Act 1965 [Ukg65]. Of key importance within the act is the requirement for all nuclear installations to be licensed and that…”the licensee shall make and implement adequate arrangements for the production and assessment of safety cases consisting of documentation to justify safety during the design, construction, manufacture, commissioning, operation and decommissioning phases of the installation.” This application of a safety case clearly covers the operation phase as well as the design and manufacture phases. One of the requirements of nuclear safety cases is the need for the licensee to demonstrate that it … “understands the hazards associated with its activities and how to control them adequately”. In July 1988, a fire on the Piper Alpha oil rig in the North Sea claimed one hundred and sixty seven lives [Ukd90]. Piper Alpha produced both oil and gas and at its peak was responsible for some ten percent of the UK’s North Sea oil production providing a strong imperative to keep operations running. It was connected to two other rigs, Claymore and Tartan, located 200km north-east of Aberdeen in the Piper oilfield. The disaster was caused primarily due to poor procedural control and a lack of safety oversight. A maintenance task was being performed on one of two gas pumps, in which the pressure relief valve was removed for overhaul. The oncoming shift should have been made aware that the pump without the pressure relief value was out of service. However, due to procedural failings, the out of service pump was used, resulting in a significant build up of pressure, pipe failure and a catastrophic explosion when leaked gas ignited. The problem was exacerbated by the sister rigs (Claymore and Tartan), who continued to pump gas to Piper Alpha via a series of underwater pipes. A key recommendation of “The Public Inquiry into the Piper Alpha Disaster” [Ukd90], chaired by Lord Cullen was that “the operator or owner of every offshore installation should be required to prepare a safety case and submit it to HSE for acceptance.” This resulted in the introduction, in 1992, of an additional set of regulations the Offshore Installations (Safety Case) Regulations (OSCR). These had the aim to “reduce the risks from major accident hazards to the health and safety of the workforce employed on offshore installations or in connected industries” [Hse06] . The MOD mandates the construction of a safety case for an equipment project during the acquisition of defence equipment and that includes manned aircraft and UAVs. These are subject to the provisions of Defence Standard 00-56 Issue 4 [Mod07b]. The guidance material contained in part 2 of the standard includes definitions for: the Safety Case (SC)and the Safety Case Report (SCR) which are discussed further below: Safety Case (SC): “A structured argument, supported by a body of evidence that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given operating environment.” [Mod07b]. The University of York Module in Hazard and Risk Management and Safety Cases [Hrm07], contends that a safety case requires two elements: supporting evidence and a high level argument. Supporting evidence comprises the results of observing, analysing, testing, simulating and estimating the properties of the system from which levels of safety performance can be inferred. A high level argument is required to provide an explanation of how the supporting evidence can be interpreted as indicating the achievement of acceptable levels of safety for the system in its operating context.
Page 19
One of the identified requirements from the OSCR, Regulation 12 [Hse06] was that the safety case must demonstrate that the management system is sufficient to ensure that relevant statutory arrangements are complied with. A further requirement is that all hazards with the potential to cause major accident (i.e. loss of life) have been identified. In the case of UAVs there will be particular aspects of both of these argument approaches that will require the provision of suitable evidence of compliance. It should be expected that many aspects will be similar to manned aircraft such as basic airworthiness and structural integrity. However there will be key differences due to the lack of the pilot being present in the vehicle. For example there will need to be arguments and evidence about “sense and avoid” as opposed to “see and avoid”; and, ensuring sufficient situational awareness to enable appropriate responses to avoid collisions. Safety Case Report (SCR): “A report that summarises the arguments and evidence of the Safety Case, and documents progress against the safety programme.” [Mod07b]. It is usual for the SC of a complex system to contain a vast amount of data (e.g. design, trials, and analysis and test data) that is generated throughout the project life cycle and produced by multiple organisations and subject to review, independent assessment and maintenance. There is likely to be multiple cross referencing making it difficult for one person to judge if the required safety levels have been achieved in the system. The SCR addresses this problem by summarising the key parts of the SC and referencing all the supporting evidence in a clear and concise manner.[Mod07b], [Hrm07]. It effectively forms the working documentary evidence against which the safety of the system can be judged at a particular point in the programme. Thus the SCR is likely to provide the quickest and most efficiently assimilated overview about system safety and is used by project team members, and others, as a working document for guidance about the safety of the system. The SCR represents a complex argument by justifying a top level claim, for example, that the system is safe to operate in an environment by suitably qualified and experienced personnel (SQEP) acting in accordance with prescribed operating procedures. The thread of the argument linking evidence to the top level claim(s) can often be obscured by the complexity and volume of text. Consequently in order to make the safety argument comprehensible, Kelly devised the Goal Structuring Notation (GSN) which has found application in many industries including military and civil aviation. [Kel99]. The GSN has significant advantages in that it is simple, structured, hierarchical and expressive providing clear communication on elements most important for safety. It can be used at various stages of argument development, and the semantics are well developed and understood. The main disadvantages are that there is a learning curve to follow to achieve a sufficiently competent standard at writing in the notation and that it can not prevent bad arguments being written. However, the notation has been expanded to offer a modular based approach so that it can be applied to safety cases for integrated modular avionics (IMA) for example and system of systems applications. Safety cases expressed in GSN can be reviewed and annotated with symbols to record objective assessments of weaknesses or strengths of the safety argument. [Hrm07]. In brief the GSN notation description is as follows: GSN shows how Goals are broken down into sub-goals supported by Evidence (solutions) making clear the Strategies adopted, the rationale for the approach (Assumptions and Justifications) A/J and the Context in which Goals are stated.
Page 20
2.2.2 The Requirements for Operational Safety Cases The examples of safety cases from the nuclear power and offshore oil and gas industries showed that these industries are regulated in order to ensure effective operational safety management. However, in other industries sufficient focus on the safety of operations does not always appear to have been sufficient, as the following example demonstrates. On 6th March 1987 the Zeebruge ferry disaster occurred with the loss of the Herald of Free Enterprise and 193 lives. This disaster occurred primarily due to deficiencies in operational safety management. The ship design was known to be susceptible to capsize. On the day of the disaster the ship left port with the bow doors open. This had become accepted practice in certain ferry operations in order to clear fumes from the car deck with the doors normally closed soon after leaving the dock. However the bosun responsible for closing the doors had fallen asleep and there was no indication in the wheelhouse of the status of the doors being open. The ship departed nose down in the water as the bow ballast tanks were still being pumped out. The Captain had written memos regarding the need for indication of bow doors but no action had been taken. Even though the ship design was not sufficiently safe, the operator must cope with deficiencies and could have invested in changes to improve the overall safety performance in operation [Her09]. In the rail industry, the yellow book [Yel05] summarises the requirements of the railway safety case regulations. Any train or station operator must write a railway safety case and have it accepted before starting operations and the operator must follow their safety case. The railway safety case must describe among other things: the operator’s policy and arrangements for safety management, how it will monitor safety, how the operator is organised to carry out its safety policy, and how the operator ensures that staff are competent to do safety related work. The definition of a Safety Case is focused on the equipment being safe for a given application and environment. If the environment or the application changes then the safety case will not be valid for the present state of the system. It is essential to keep the safety case under review and ensure that any changes made to the system design and operational arrangements deliver the required levels of safety [Mod07c], [Yel05], and [Omm08]. Previous post graduate students at the University of York have investigated OSCs: Blagrove [Bla04], Salter [Sal06] and Jones [Jon07]. Blagrove developed a “prototype generic operational safety case pattern for possible reuse in the UK military aerospace domain” [Bla04]. His work