18a evaluation of pps - sandia.gov18 - evaluation of physical protection systems the twenty-sixth...

18 - Evaluation of Physical Protection Systems

The Twenty-Sixth International Training Course

18 . In t roduc t i on to theEva lua t i on o f Phys i ca l P ro tec t i on Sys tems

October 24 – November 11, 2016Albuquerque, New Mexico, USA

Riyaz Natha

Introduction to Evaluation of PPS

Learn ing Object ives

After completing this module, you should be able to:• Identify the physical protection system effectiveness

measures: Probability of Interruption, PI

Probability of Neutralization, PN

• Recognize two basic PPS evaluation methodologies• Describe the set of tools used in ITC to evaluate the

system effectiveness of the hypothetical facility• Define factors that affect evaluation quality

2




INFCIRC/225/Rev.5 - Evaluat ions

• To ensure that physical protection measures are maintained in a condition capable of meeting the State’s regulations and of effectively responding to the State’s requirements for physical protection, the State’s competent authority should ensure that evaluations based on performance testing are conducted by operators at nuclear facilities… (3.21)

• Evaluations, including performance testing of the physical protection measures and of the physical protection system, including timely response of the guards and response forces, should be conducted regularly to determine reliability and effectiveness against the threat. (4.35 and 5.41*)

*4.35: Requirements for Categories I and II nuclear material5.41: Sabotage of high consequence facilities including nuclear power plants

3


Recommendat ions for Eva luat ions o f Phys ica l P rotec t ion Sys tem Des igns to Prevent Sabotage

• The physical protection system should be designed to deny unauthorized access of persons or equipment to the targets, minimize opportunity of insiders, and to protect the targets against possible stand-off attacks consistent with the State's threat assessment or design basis threat. (5.14)

• The operator should evaluate and the competent authority should validate the design of physical protection system effectiveness to verify that it complies with the required level of protection for the nuclear facility and nuclear material. (5.15)

• Using the threat assessment or design basis threat, the operator – in cooperation with the State’s competent authority - should define credible scenarios by which adversaries could carry out sabotage of nuclear facilities and nuclear material. (5.9)

4




ITC Des ign and Eva luat ion Process Out l ine (DEPO)

Define PPSRequirements

DesignPPS

Evaluate PPS

Final PPSDesign

RedesignPPS

5


Performance-based Approach

• Recall, from the risk management and regulatory requirements module, the performance approach: Competent Authority specifies the required level of performance

against the DBT Operator complies by designing and evaluating its physical

protection system to achieve the required performance level Competent Authority is responsible for verifying that the

Operator’s system satisfies the required performance against the potential adversary

6




Eva luat ion Object ives

• Competent Authority and Operators have complementary objectives for the evaluation of PPS: Meet regulatory and operator requirements

• Self-assessment by operators• Inspection by competent authority• Periodic re-validation

Verify and/or improve PPS performance• Verify PPS satisfies PE and design requirements• Identify system deficiencies• Analyze system upgrades• Compare cost versus performance• Select / implement overall best option

7


Performance Evaluat ion Metr ics

• Three metrics are commonly used for the evaluation of the performance of PPS: System Effectiveness (PE) Probability of Interruption (PI) Probability of Neutralization (PN)

8




System Effect iveness• One method to determine PE is to treat PI and PN as

independent variables, and use the following formula:System Effectiveness (PE): The probability that the physical protection system will prevent the adversary from completing the undesired event

PE = PI * PN

• Other methods Establish minimum performance values for PI and PN

Use results of overall scenarios to determine PE without independently calculating PI and PN

• In this method, PI is used as an additional performance measure

Recall that we introduced the terms and concepts of interruption and neutralization during the introduction to the design section of ITC DEPO

9


Two Methodo log ies Address D i f fe rent Aspects o f PPS Ef fec t iveness

• Path Analysis Does the PPS design adequately provide:

• Timely detection?• Defense in depth?• Balanced protection?

• Scenario Analysis Does the PPS design provide the required level of protection

against an adversary attack (scenario) consistent with the Design Basis Threat?• For this course, protection requirements are in terms of PE being

above some threshold, such as 85%

10




Des ign Adequacy

• Adequacy is measured in terms of Probability of Interruption (PI) along a path PI is based on adversary and response timelines

• Design adequacy is evaluated using two criteria: Timely detection and balanced protection: Is PI high against the

DBT along all adversary paths? Defense in depth: Are there diverse detection and delay

measures along all potential adversary paths?

11


Concepts Assoc ia ted w i th Probab i l i ty o f In ter rupt ion: Adversary and PPS T ime l ines

Detection Time

AdversaryBegins Task

Adversary Completes Task

Time

Adversary Task Time

CT

FirstSensing

T 0

Ad

vers

ary

Det

ecte

d

DT

Response Force Time

Ad

vers

ary

Inte

rru

pte

d

T I

PPS Response Time

Adversary Task Time Remaining After First Sensing

Sensing Opportunities

Time Remaining

After Interruptio

n

12




P r inc ip le o f T ime ly Detec t ion and Cr i t i ca l Detec t ion Po in t

Principle of Timely Detection: To interrupt the adversary before the theft or sabotage task is completed, the PPS response time must be less than the adversary task time remaining after the first sensingCritical Detection Point (CDP): The last sensing opportunity along an adversary path for which the PPS response time is less than the adversary task time remaining after the first sensing

13


Probabi l i ty of Interrupt ion Def in i t ionProbability of Interruption: The cumulative probability of detection along a path timeline up to and including the CDP• Here, the cumulative probability combines probabilities of detection

PD1 and PD2 at the two sensing opportunities

14




Another V iew of Probabi l i ty of Interrupt ion

• PI is the estimated probability that timely detection will occur during an attack represented by the adversary timeline

15


Pa th Ana lys i s Uses Adversary Sequence D iagramsAdversary Sequence Diagram (ASD): A graphical model used to help evaluate the effectiveness of the PPS at a facility

16




Adversary Path i s Def ined on Adversary Sequence D iagramAdversary Path: A time-ordered sequence of path elements, areas, and a target task that the adversary must traverse to complete an attack from offsite to the chosen target

17


Purpose of Path Interrupt ion Analys is

Path Interruption Analysis: Determines whether detection and delay are sufficient along all adversary paths to provide an adequate level of Probability of Interruption, PI, based on planned PPS Response Times

18




Probabi l i ty of Neutra l izat ion

Probability of Neutralization (PN): The probability that the response force can prevent an adversary from completing a malicious act such as theft of nuclear material or sabotage of a nuclear facility

Response force must neutralize adversary following interruption Neutralize means response force either arrests, captures, or kills

adversary, or causes adversary to flee

19


Neutra l izat ion Analys is Methods

• Expert judgment• Simple numerical methods for PN

Typically for path analysis Determines PN for a path

• Simulations Scenario analysis Determines PN as part of PE

• Actual engagements

20




What Is Scenar io Analys is?

Scenario Analysis: A methodology for analyzing system effectiveness, PE, by considering several alternative possible adversary attacks (scenarios)

Allows more detailed analysis of the attack, the defense, and the results than path interruption analysis

21


S imulat ion Techniques

• Several simulation techniques are used in scenario analysis to estimate PI or PN, or PE Structured and detailed tabletop exercise Computer simulation of small force engagement Force-on-Force exercises

• Performance test results are used as input to all simulation techniques Security equipment tests Limited scope performance tests

22




What is a Tabletop Exerc ise?

Tabletop exercise: A method to simulate an adversary attack on a site’s existing or proposed Physical Protection System (PPS)• Analyzes PPS elements

Detection Delay Response

• Provides insight into a PPS that can stand alone or be used in other analysis tools

Evaluation Team

Protective Fo

rce

Adversary Team

ExerciseModerator

SiteMap

23


Eva luat ion Methodo log ies for PPS Ef fec t iveness

• Two complementary methodologies are typically used in the evaluation of PPS effectiveness Path Modeling and Analysis

• Path interruption analysis determines whether detection and delay are sufficient along all potential adversary paths to provide an adequate level of Probability of Interruption (PI), based on planned response times

Scenario Development and Analysis• Scenario analysis determines whether the PPS effectiveness, PE, is

adequate across the range of detailed attack scenarios that might be credibly planned and conducted by adversaries within the Design Basis Threat

24




Current Evaluat ion Best Pract ice

• Use a combination of Path Interruption Analysis

• One of several analytical tools

Scenario Analysis• Several simulation techniques

• Look for consistency among results

25


ITC Evaluat ion Process for Outs ider

• Because we are limited (time, resources) in our ITC training environment, we use the following process to estimate PPS effectiveness against outsider threats: Probability of Interruption (PI)

• Analytical path timeline model

Probability of Neutralization (PN)• Simple numerical model

Probability of System Effectiveness (PE)• Quantitative: PI * PN

• Qualitative: Tabletop exercise

26




ITC Evaluat ion Tools

• We will use a few illustrative evaluation tools in ITC to gain understanding of the evaluation process: Equation: Cumulative probabilities of detection

• PI for analyzing a single adversary timeline and response timeline

MP VEASI software: Multi-Path Very-simplified Estimate of Adversary Sequence Interruption• PI for multi-path analysis

Table based on numerical model: variables for force sizes• PN for small force engagement

Tabletop exercise: Picture in time• PE for user-defined scenario

27


ITC Evaluat ion Modules

• The Evaluation Section of ITC DEPO consists of nine modules: Adversary Sequence Diagram (ASD) Path Interruption Analysis (PI) Multipath Analysis (MP VEASI) Neutralization Analysis (PN) Scenario Analysis (PI) Tabletop Analysis Insider Analysis Transportation Security Information Security

28




Ins ider Analys is - System Approach

29


Transportat ion Secur i ty Analys is

• We introduce transportation security design and evaluation and compare it to facility security Similar DEPO process

• Requirements, design, evaluation

Transport DBT – may be different from facility DBT Design implementation – transport uniqueness

• Detection – by escorts• Delay – by transport vehicle• Response – by escorts

Evaluation – primarily by scenario analyses• Tabletop exercise, computer simulations, force-on-force

30




Informat ion Secur i ty

• We use the DEPO process for information security

Finalize System

Revise System

Define Requirements

Design System

Evaluation System

— Targets

— Threats

— Regulator

— Administrative Controls

— Technical Controls

— Mitigation and Recovery

— Performance Measures

— Evaluation Methodologies

31


Eva luat ion Qual i ty

• Two major factors determine the quality of the PPS performance evaluation Subject matter experts

• Subject matter experts and their expert knowledge and experience are involved in the application of all evaluation methodologies

Performance test data• Security component (detection, delay, and response) performance

data used in the system evaluation must be high quality• Component performance data should be based on current

performance testing

32




Summary

• The three PPS performance metrics used in ITC are PI, PN, and PI

• The two basic PPS evaluation methodologies Path analysis Scenario analyses

• PPS evaluation tools used in ITC PI equation, MP VEASI, PN numerical model, Tabletop exercise,

and insider analysis

• Two major factors affecting evaluation quality Subject matter experts Performance test data

33

18a evaluation of pps - sandia.gov18 - evaluation of physical protection systems the twenty-sixth...

Documents