9th annual secure and resilient cyber architectures ... · renamed “invitational” and held in...

1 ©2020 The MITRE Corporation. All Rights Reserved Approved for Public Release, Distribution Unlimited. Case # PR 19-02172-19

9th Annual Secure and Resilient Cyber Architectures Invitational & Training Event: 2019 Proceedings


Table of Contents Overview ........................................................................................................................................ 4

Background ................................................................................................................................... 4

Prior Years: 2010 - 2018 ............................................................................................................. 4

This Year: 2019 ........................................................................................................................... 5

Introduction ................................................................................................................................... 6

Section 1: Tutorial, Wednesday, Sep 4th ..................................................................................... 8

Section 2: Presentations, Thursday, Sep 5th ............................................................................... 8

Welcome ..................................................................................................................................... 8

Kick-Off Talk Cyber@MITRE ................................................................................................... 9

Operating in a Cyber Contested Environment .......................................................................... 10

Mission Metrics in Support of Resilient System’s Architecture .............................................. 11

New Paradigms for the Next Era of Security ............................................................................ 12

Deception .................................................................................................................................. 13

Section 3: Presentations Friday September 6th ........................................................................ 15

Cyber Security in the Age of Connected Medical Devices ...................................................... 15

Cyber Resiliency in the Energy Sector ..................................................................................... 17

Use of Analog Measures to Provide Cyber Resiliency in the Energy Sector ........................... 18

Panel: Cyber Resiliency in the Financial Sector ....................................................................... 20

NIST SP 800-160 Volume 2, Final Public Draft, Developing Cyber Resilient Systems – A systems Security Engineering Approach ........................................................................ 22

Product Vendor Talks ................................................................................................................... 24

Panel: Compare and Contrast – Cyber Resiliency Issues Across Domains .............................. 24

Section 4: Break Out Sessions, Thursday, September 5th ....................................................... 25

Breakout Session: Cyber Resiliency Metrics, Measurements, and Assessments ..................... 25

Goal .................................................................................................................................. 25

Discussion / Observations ..................................................................................................... 25

Recommendations/Way Forward .......................................................................................... 28

Breakout Session: Cyber Resiliency Table-Top ....................................................................... 28

Goal .................................................................................................................................. 29


Background ............................................................................................................................... 29

The Scenario ............................................................................................................................. 29



Break out Session: Deception ................................................................................................... 32

Goal .................................................................................................................................. 32



Break out Session: Cyber Resiliency in Weapon Systems ....................................................... 34

Goal .................................................................................................................................. 34


Issues, Challenges or Outcomes ........................................................................................... 35


Overview September 2019 marked the ninth year in which approximately 170 subject matter experts (SMEs) in cyber resiliency from government, industry, and academia came together in McLean, VA, for collective work on topics of common policy and engineering concern. For two days, the 9th Annual Secure and Resilient Cyber Architectures Invitational & Training Event accelerated recognition and adoption of cyber resiliency with a focus on organizations.

Background Prior Years: 2010 - 2018 The first workshop, held in October 2010, established the initial community and shared architectural, technical, and policy perspectives on cyber resiliency. The second workshop, held in May 2012, focused on collaborating to develop a communal view of resiliency frameworks, engineering principles, and metrics [1]. The third workshop, held in June 2013, centered on identifying favorable conditions for use of specific resiliency techniques, assessing the use of techniques in enterprise architectures, and developing use cases [2]. The fourth meeting, now renamed “Invitational” and held in May 2014, emphasized applying cyber resiliency to space-based systems and critical infrastructure, designing a cyber resiliency challenge, and identifying roles played by cyber resiliency throughout the systems engineering life cycle [3]. The Fifth Annual Secure and Resilient Cyber Architectures Invitational, held in May 2015, concentrated on taking stock of the state of cyber resiliency: the lessons learned and the remaining challenges to overcome. It sought community consensus on the theme of Cyber Resilience: Looking Backward (What Has Worked? What Has Not?), Looking Forward (What New Challenges Must Be Faced?). Keynote speakers included representatives from the National Institute of Standards and Technology (NIST), US Navy, Indiana University, and Bit9 + Carbon Black [4]. The Sixth Annual Secure and Resilient Cyber Architectures Invitational, which took place in May 2016, centered on the theme of Institutionalizing Cyber Resiliency [5]. Four keynote speakers were followed by panel discussions inclusive of industry leaders. Three working groups furthered knowledge sharing by focusing on: cyber resiliency and system security engineering cyber resiliency and an organization’s cybersecurity program, and cyber resiliency and acquisition. In addition, vendor booths and representatives displayed leading-edge cyber resiliency offerings.

The Seventh Annual Secure and Resilient Cyber Architectures Invitational and Training Event was held in May 2017. An optional tutorial was offered prior to the Invitational, the addition of the tutorial was the reason that that “Training Event” was added to the title. The event included four presentations, a panel, four facilitated working groups, and selected vendors booths and presentations by those vendors. Topics addressed included: cyber resiliency in the financial community, cyber resiliency and architectures, measuring the effectivenss of cyber resiliency, and cyber resilience in weapon systems.


The Eighth Annual Secure and Resilient Cyber Architectures Invitational and Training Event was held in May 2018, the event included thirteen presentations, a lightning session, five three-hour breakout sessions, and vendor booths representing cyber resiliency related products. In addition, there was an optional tutorial the afternoon prior to the commencement of the event. Topics addressed included: cyber resiliency of weapon systems, operationalizing resiliency, micro-virtualization, resiliency in elections, and cyber resiliency without detection.

This Year: 2019 The rest of this report focuses on the Ninth Annual Secure and Resilient Cyber Architectures Invitational and Training Event. These proceedings present a summary of the keynote talks, the panel discussion, and working group tracks. The Cyber Resiliency Invitational Committee believes the Invitational serves a larger mission: to advance the field of cyber resiliency for our sponsors and nation. Additional materials from the invitational and briefings can be found at https://www.mitre.org/cyberworkshop. The committee welcomes comments from readers through the contact email address: [email protected].

The Cyber Resiliency Invitational Committee September 2019


Introduction The Ninth Annual Secure and Resilient Cyber Architectures Invitational & Training Event included seven presentations, two panel discussions, vendor product briefings, and four three-hour breakout sessions. In addition, there was an optional tutorial the afternoon prior to the commencement of the event which is discussed briefly in Section 1.

Section 2 summarizes the following presentations provided on Thursday September 5th:

• Introduction by Dr. Vipin Swarup, MITRE, Director Cyber Resilient Systems • Invitational Kick Off by Mr. Gary Gagnon, MITRE, VP Cyber Security and Chief

Security Officer • Operating in a Contested Environment by Mr. John Garstka, Director, Cyber, Office of

the Chief Information Security Officer, Office of the Undersecretary of Defense for Acquisition and Sustainment

• Missions Metrics in Support of Resilient Systems Architecture by Dr. Robert Lychev, Technical Staff, MIT Lincoln Laboratory

• New Paradigms for Next Era of Security by Mr. George Roelke, MITRE, Cyber Innovation Area Lead

• Deception by Dr. Stan Barr, MITRE, Senior Principal Cybersecurity Engineer

Section 3 summarizes the following presentations and panel discussions provided on Friday September 6th:

• Cyber Security in the Age of Connected Medical Devices by Ms. Debra Bruemmer, Mayo Clinic, Senior Manager, Clinical Information Security – Resiliency, Office of Information Security

• Cyber Resiliency in the Energy Sector by Mr. Fred Hintermister, Global Resilience Federation, VP for Energy Protection

• Use of Analog Measures to Provide Cyber Resilience in Energy Sector by Mr. Brad Stephenson, SouthernCo, Senior Manager, Advanced Threats, Intelligence, and Deterrence

• Panel: Cyber Resiliency in the Financial Sector with o Mr. Mark Morrison, OCC, SVP & Chief Security Officer / Security Services o Mr. Mark Brubaker, Bank of America, SVP, Cyber Resilience Executive o Mr. Jerry Archer, Sallie Mae, Chief Security Officer

• NIST SP 800-160 Volume 2 by Dr. Ron Ross, NIST Fellow • Cyber Resiliency Product Presentations • Panel: Compare and Contrast – Cyber Resiliency Issues Across Domains with

o Milos Manic, Ph.D., Director, VCU Cybersecurity Center Affiliate, Idaho National Laboratory (INL)

o Mr. Zach Furness, Director of IT Security, INOVA Health Systems o Mr. Bob Bigman, President of 2BSecure


Section 4 summarizes the following four breakout sessions provided during the afternoon of the September 5th:

• Cyber Resiliency Metrics, Measurements, and Assessments o The definition and evaluation of metrics to assess the effectiveness of cyber

resiliency mitigations continue to present significant challenges. There is no single agreed upon best cyber resiliency set of metrics or means of measuring the effectiveness of cyber resiliency solutions. Approaches vary in terms of the level of abstraction (e.g., mission, system), target stakeholder (e.g., mission owner, cyber defender, architect), how assessments are determined (e.g., SME driven, model-based, laboratory-based). This session included discussions and presentations highlighting various measurement / metric methodologies from different speakers and organizations.

• Cyber Resiliency Table-Top

o The session included a series of table-top exercises based on real-world cyber-attacks, centered around an actual cyber-attack on a country’s energy sector. It focused on the cyber-attack on the Ukraine power grid a few years ago, viewed through the prism of ATT&CK for Industrial Control Systems. The table-top employed the Maelstrom cyber-attack life cycle game to help examine how applications of cyber resiliency techniques and approaches identified in NIST SP 800-160 V2 might have potentially mitigated the cyber-attack.

• Cyber Resiliency in Weapon Systems

o The session drew upon recent work to examine challenges of and approaches to applying cyber resiliency in weapon systems. The session looked at work being done to develop a cyber resiliency framework for weapon systems, including associated principles and a questionnaire to assess whether principles have been applied. Due to the nature of the topics discussed, this session was limited to US government personnel and their FFRDCs, National Labs, UARCs, and contractors.

• Deception

o The session discussed if and how deception can be used to support cyber resiliency, especially in enabling cyber deterrence and adversary management. It examined some of the technical challenges or innovations needed to improve the effectiveness of cyber deception technology across multiple industries and businesses of all sizes. Some questions for the discussion included: What technical requirements would one need to get started using deception? What does a well-built deception environment look like, and how hard is that to achieve? How long do I want an attacker in a deception environment? If I already know which APTs are targeting me; why would I use deception to manage my adversary?


In parallel to the presentations and breakout sessions, during both days there were cyber resiliency related product booths. The booths were represented by: Air Force Research Laboratory (AFRL), Attivo Networks, GuardiCore, Illumio, Illusive Networks, Nozomi Networks, Objective Interface Systems (OIS), Polyverse, and Tripwire.

Section 1: Tutorial, Wednesday, Sep 4th

A tutorial on Cyber Resiliency: Introduction and Recent Developments was presented by Ms. Deb Bodeau, Mr. Rich Graubart, and Ms. Rosalie McQuaid, of the MITRE Corporation and Mr. Jim Reilly of AFRL. An outline of the tutorial is presented below:

Section 2: Presentations, Thursday, Sep 5th Welcome Dr. Vipin Swarup, Director, Cyber-Resilient Systems, The MITRE Corp.

Dr. Swarup, served as the master of ceremonies for the Invitational. He provided an overview and look back, including MITRE’s 50 year’s working cyber security and nine in cyber resiliency. The purpose of the event, as stated by Dr. Swarup, was:

• To bring the community together to work collectively to accelerate recognition and adoption of Cyber Security Resilience, and

• To recognize that achieving success in cyber resiliency is beyond any one organization or sector, and success in cyber resiliency requires collaboration across government, industry, academia, and FFRDCs.

Dr. Swarup reviewed the varied presenters of the Invitational for the past years (listing those from government, industry, academia, and FFRDCs/National Labs) and he provided a concise summary of the progression of cyber resiliency over the past eight annual events as well as the impact of the Invitational, as seen in his chart below.

Time Contents 1:30-3:00 Cyber Resiliency Overview

• Introduction • Motivation and Terminology • Example Scenario • Introduction to the Cyber Resiliency

Engineering Framework (CREF) • Cyber Resiliency Solutions

• Building Cyber Resilient Systems 3:00-3:15 Break

3:15-4:15 The AFRL Cyber Survivability Attributes (CSA) Tool

4:15-4:30 The CREF Navigator


Kick-Off Talk Cyber@MITRE Mr. Gary Gagnon, MITRE’s Vice President, Cybersecurity and Chief Security Officer, The MITRE Corp.

The kick-off briefing was to be given by Mr. Gary Gagnon, MITRE’s Vice President, Cybersecurity and Chief Security Officer. Unfortunately, due to illness he was not able to attend, and his briefing was presented by Mr. Chris Folk, MITRE’s Director of Cyber Strategy. Chris talked about MITRE’s mission to secure cyberspace for a safer world. He noted that MITRE’s North Star was “Achieving measurable improvements in cyber resiliency to our critical infrastructures.”

He explained that in 2009, just before Christmas, MITRE had a wakeup call –an APT in MITRE’s system. MITRE had seen indicators of this APT before – and it was not going away – and MITRE realized that it could not be kept out. So instead MITRE decided to let them stay in. MITRE’s approach was to engage the APT and learn what they can do in a controlled way, with the goal that MITRE could learn more from the APT than it got from MITRE.

Chris went on to say that MITRE’s objectives with regards to cyber were:

• Build communities and strategic partnerships focused on strengthening security and resilience of our nation’s critical cyber systems and infrastructure.

• Deliver innovative solutions and toolkits that change the way industry approaches cybersecurity and privacy.

• Develop new business models, language, and operational capabilities that enable broader impact in cyber and disrupts the precepts held by industry about hording data to enabling them to deliver solutions.

Finally, he discussed the MITRE Cyber themes for the Invitational:

• Metrics and Measurements • Weapons systems and Critical Infrastructure • Deception


The themes were the focus of the break-out sessions.

Operating in a Cyber Contested Environment Mr. John Garstka, Director, Cyber, Office of the Chief Information Security Officer, Office of the Undersecretary of Defense for Acquisition and Sustainment

Mr. Garstka began by noting how one of the goals of the DOD cyber Strategy 2018 is ensuring that the Joint Force can achieve its missions in a contested cyberspace domain. He went on to explain that cyberspace is a contested environment that has an impact on both adversary and defender kinetic forces. He also described how the US Forces operating in Cyberspace are organized at both the tactical and strategic levels.

He went on to explain that while there is a range of adversaries that the DoD encounters, that the DoD must be able to operate against Tier IV adversaries. These are known as “5-D adversaries”, as their goals are to deny deceive disrupt degrade and destroy defender mission operations.

Mr. Garstka talked about the concern that one is never as invincible as one believes. To support this view, he shared a story that one week before the USS Arizona was sunk by the Japanese at Pearl Harbor there was a Navy training exercise where it was said that “… despite the claims of air enthusiasts, no battleship has yet been sunk by bombs.” Mr. Garstka noted that many commanders have never experienced a cyber threat and implied that they may have similar views to the pre-Pearl Harbor battleship advocates. He noted how many of today’s weapons systems built were built before the existence of cyber threats. These cyber threats need to be incorporated into defense planning - paradigm shift. You can create a cyber-attack that defeats a whole class of systems

Mr. Garstka discussed how cyber-attacks are not simply an attack on systems and networks. Rather such attacks can impact an entire mission and an organization’s ability to operate. He noted how it is not sufficient to secure the weapons systems themselves unless one can secure the DIB, because otherwise the adversary can keep getting in and getting all the information as well.


• He discussed the DoD Cyber Technical Direction Agent (TDA), which a joint MITRE, APL and Lincoln Lab activity effort. The cyber TDA builds on existing strategies for understanding a system in terms of resilience:

– Identifies foundational resilience/hardening principles for implementation – Provides guidance on developing mission-based measures of effectiveness for these

principles – Introduces an evaluation methodology for understanding enhanced resilience – Framework validation through experimentation will show the impact of these

foundational principles on mission and system performance.

Mission Metrics in Support of Resilient System’s Architecture Dr. Robert Lychev, Technical Staff, MIT Lincoln Laboratory

Dr. Lychev began by noting that we live in a continuously cyber-contested environment. The threat to our mission systems is real, and it is constantly evolving. He went on to note that today’s weapon systems are becoming more software intensive and more complex. He noted for example that the F35 has over 8 million lines of code, and that does not include the logistics elements.

He noted how a representative weapon system consist of different components: people (operators, analysts, decision makers) and different actuators (e.g., fighters, recon planes), many built by different vendors. These two parts are connected via software communications systems. The software is constantly being updated, and it is so complex that no one truly understands it. This leads to complexity and unpredictability.

There are many ways an adversary can gain access to the weapon systems including compromised users, hardware, software, and input/output devices. The threats and ability to compromise such systems only increase with the complexity of them.

He then offered an occurrence plausibility vs. impact model. The goal is to reduce both factors. To do that requires understanding of the contributing stress factors and consequences they may have, and then prioritize them.


To do this he offered up a cyber resilient life cycle with nine categories. He then talked about how to put the life cycle process into practice. He did note that complex systems are intrinsically unpredictable. Despite that he offered that we could represent the system through a combination of a state model and component model. The first step is to decompose the system into mission essential functions. As part of the decomposition process various factors are considered including quality, timeliness, and cost. In the case of an aircraft additional factors may be considered including throughput, availability, load, latency, errors and forensics.

The next challenge is how to collect all the necessary data. He noted that the information should be collected as close to the source as possible to maximize trustworthiness. Dr. Lychev then suggested practicing failure and attacks – using chaos engineering needs to be done in a controlled fashion in weapons system – i.e., practice critical times during non-critical times. He then used a few slides to illustrate via a hypothetical scenario where he deliberately corrupts aircraft logistics data.

New Paradigms for the Next Era of Security Dr. George Roelke, MITRE, Cyber Innovation Area Lead

Dr. Roelke talked about how the attacks by adversaries are evolving with new attack vectors such as side channel analysis and supply chain attacks. AI is a new challenge along with machine learning; adversaries are going to use it more, thus providing greater autonomy of attacks. From the defender side there are challenges with regards to operations, intelligence, requirements, R&D, and test and evaluation. These challenges require confronting questions such as:

• How does a defender forecast cyber threats across decades long operational life of the system?

• How does a defender create clear requirements against realistic threats? • How does one make design trade-offs such as between hardening nodes and adding

redundant nodes or links to be more resilient to attack? • Test and Evaluation Challenges: What threats does one test against? How does one do

testing against autonomous systems? • How do we measure success? • Adversary work factor: How much time will it take an adversary to develop and exploit

– need to collect information on capabilities of adversary whether they are foreign or criminal?

Dr. Roelke noted how more organizations are employing autonomous systems to aid in their defense. That enables cyber analysts to more rapidly interpret cyber observables and determine the best course of action to protect missions from adversaries. But at the same time this causes the autonomous system defender systems to themselves be targets of the adversary, often via deception activities.

He discussed CALDERA – a MITRE-developed, automated red teaming tool. It uses artificial intelligence concepts in goal-directed planning and the MITRE ATT&CK enterprise network adversary model to automate adversary emulation and perform network assessments, testing, and


training in a repeatable, cost-effective way. An open source version of CALDERA is web accessible.

Dr. Roelke also discussed the challenges of measuring the effectiveness of cyber resiliency. He talked about MITRE’s Measuring the Effectiveness of Cyber Resiliency (MECR) methodology. He noted how it used the cyber resiliency engineering framework of NIST SP 800-160 V2, and that the MECR activity had generated over 400 cyber resiliency metrics.

He discussed the safe testing of cyber physical systems and the use of a simulator and safety enforcer to support such testing. He also discussed operational research challenges. These include:

• How does one determine in real-time which systems are contributing to the mission –which are the key missions and what networks/systems are supporting it.

• Automating decision making – human machine learning. What information does one present to an operator; he noted that this information changes based on operator expertise level, operation, exhaustion, and similar factors.

Dr. Roelke’s closing comments were a call to action. Among the points he raised were:

• The importance of asking your research community for help • He reiterated the intelligence challenge projecting threat and adversary capability is hard

especially over the long term. • Requirements: Defining better cyber resiliency requirements and tools for prioritizing

between missions is difficult; organizations need to understand that there is not a single solution in part because a platform rarely has a single mission.

• R&D: Strive for provably secure but don’t expect to get there soon. Don’t expect perfection – particularly over a long period of time.

Deception Dr. Stan Barr, MITRE, Senior Principal Cybersecurity Engineer

Dr. Barr referred to what Chris Folk had mentioned in his talk, that at APT attack on MITRE in 2009 was a wakeup call. Stand explained that much time was spent wading through indicators from other organizations (with limited relevance to MITRE) and an overwhelming amount of CVEs. There was very little information on post-compromise TTPs.

He noted how human spies are very expensive and very fragile. The intelligence they can collect is very valuable, but it is fraught decision to use if there were only a few people in the room. Cyber is different – our adversaries can use a tool that is quick to build, can be fired at a whole class of victims and even if it is discovered by one organization it might be useful to another organization because intelligence is not shared widely why would they rely on humint or sigint when cyber is so easy and so effective? So, the challenge then is how do the defenders make the attackers question the value of cyber?


He explained how the cyber defender engagement with the adversary is a continuum. At one end organizations rely on passive things like logs and intrusion detection and some almost active things like reactive honey - these rely on Luck. The other end is Proactive – hack back (lots to talk about but not our focus – there is a lot of peril in that – someone needs to clean these things up on the network and fix the internet but we don’t have the law in place – so doing that is a bit frightening). He

argued for organizations being in the middle of the continuum– using constructs such as honey canaries to provide deception for detection.

Dr. Barr noted how many people think we can block and kill our way out of this problem. But this keeps us from learning about the adversary, but it allows the adversary to keep learning about us and doing things until they succeed (and we don’t know it). This causes them to evolve making it harder for us to know what they are doing.

He commented that while defense in depth is important, it is not sufficient. Dr. Barr cited Joint Pub 3-13.4 – Military Deception. “Deception is actions executed to deliberately mislead adversary … decision makers thereby cause the adversary to take specific actions (or inactions that will contribute to the accomplishment of the friendly mission”. Dr. Barr also cited Napoleon: don’t interrupt your adversary when they are making a mistake”

Dr. Barr offered a Dam Metaphor. He compared the adversary to an endless stream. The steam of water will find a way through. But the stream (and the adversary) follow the path of least resistance. One can use dams to channel them away from important things. Instead of expending resources on developing and deploying great zero day attacks an adversary will find unpatched systems that will allow them to be successful with little effort. Dr. Barr therefore posed the question, how do we use our system vulnerabilities to catch our adversary? How do we let them in on our terms?

Dr. Barr’s answer to that question is three part.

1. Use honeypots to provide for initial detection 2. Use tradecraft to elicit interest and IOCs 3. Manage the adversary, present them with information that matches their interest but that

you are willing to release.

Dr. Barr noted that one of the things that MITRE learned during these engagements was that the adversaries MITRE was dealing with generally had a limited set of software tools and generally chose not to modify them because the tools worked. That aided MITRE, as once MITRE discovered the tools the adversary was using, they knew how to handle and feed it.


MITRE manipulates the adversary in a synthetic deception environment similar to the actual target environment. The synthetic environment employs both virtual and physical assets, typically customized to reflect the target and the threat. The use of such an environment provides MITRE with intelligence with regards to adversary’s tactics, tools, procedures, mission, and goals (what are they after).

Dr. Barr also noted that the world is changing somewhat and that now the APT needs to be wary of engagements with defenders. There is increased awareness of the APT activities. Adversary cyber operations are vulnerable to many collection activities. Adversary threat data is becoming an international commodity. Finally, commercial threat intelligence shops now have a financial motivation for revealing adversary activities.

Dr. Barr offered up some other observations and experience. He noted how in some instances when the adversary realizes they are in a deception environment they have been known to trash it out of frustration, but assuming it is a virtual deception environment, that is really no loss to the defender. Regardless of the adversary attempting to trash the deception environment, it is beneficial for the adversary to not be aware they are in a deception environment as they are not motivated to change their TTPs.

In conclusion, deception is a valuable tool for managing the APT. The use of “polluted” documents can complicate the work of the adversary by causing them to sift through misleading and conflicting information. Not only does this cause the adversary to waste time and resources but every minute that the adversary is deceived is a minute that they are not stealing real data.

Section 3: Presentations Friday September 6th Cyber Security in the Age of Connected Medical Devices Ms. Debra Bruemmer, Mayo Clinic, Senior Manager, Clinical Information Security – Resiliency, Office of Information Security

Ms. Bruemmer began with explaining to the audience the healthcare and medical device environment.

She noted how the primary mission was ensuring safe patient care. The consequences of loss are different in the medical domain – cannot replace a patient’s medical record like replacing a credit card. She explained that while there is great awareness of the growing cyber threat to medical devices that change (response to the threat) is slow. One reason is limited finances. She noted how 70% of hospitals have less than 200 beds and that 80% of medical device vendors have less than 50 employees. Another reason for the slow rate of change is nature of the underlying software in the devices. Medical devices have a “long shelf life”. But they are often developed using commercial underlying software that has a relatively short shelf life. As an illustration she noted that many devices still use Windows XP (whose supported ended in 2014) and large number still use Windows 7 whose support ends in 2020. Finally, she noted that risk is largely managed through guidance. There are no FDA directives, medical device vendors have little incentives to add security and there are very limited consequences for selling devices that lack adequate security.


As Ms. Bruemmer explained, the target list of technical assets supporting medical decisions is broad. It ranges from traditional networks, WiFi, servers, and workstations, to applications, IoT devices (e.g., refrigerators), to specialized medical devices (e.g., infusion pumps, MRIs). This diverse set of IT poses a problem for hospitals regarding security. For more traditional systems (e.g, networks) the hospital can employ concepts such as segmentation or NAC to provide some security. But for specialized medical devices only the vendor can make changes, changes by hospital would invalidate the warranties.

She provided the audience with Mayo Clinic’s Medical device philosophy:

o Pushing security into the procurement process. But this is somewhat tempered because security is not the last word as the physicians lead the procurement selection

o Manage legacy device fleets o Test medical devices internally in secure/segmented area; do not wait for vendors. o Document and share findings with vendors – frequently they are unaware of the

problems. This approach is better than public disclosure o This approach ends up benefiting society – because Mayo is encouraging vendors

to push security fixes out to all the products

Mayo is trying to improve security through a series of practice measures:

• Set security standard: Mayo has developed 77 requirements built on 19 ISO capabilities; philosophy is to enhance what exists (ISO standard)

• Set minimum security requirements: If vendor product does not meet them then ask them to fix it. Give them a timeline and commitment to fix it. Exceptions are handled through a governance group, which centralizes risk management.

• Evaluate new purchases: focus on the high priority devices and be sure to assess the whole device family; but emphasis is on patient safety.

• Security requirements in contract: Review vendor security program. Comply with FDA guidance. Vendor performs necessary testing and scanning

• Remediation actions: Remediation is challenging as most require vendor action. Some can be done by hospitals (e.g., isolate devices that do not comply with all security requirements) but are resource intensive.

• Governance of Risk: Establish Mayo Clinic Security Committee. Identifies security risks and develop Mayo wide strategies and policies to deal with them. Committee is chaired by physician. As result of the committee, Mayo no longer allows the individual departments to make risk decisions that impact the entire organization.

Ms. Breummer also spoke about over the horizon concepts, in particular, Clinical Workflow Resiliency. The vision of the Clinical Workflow Resiliency is to enhance existing business processes to enable ongoing delivery of patient care for defined critical clinical workflows during a cybersecurity event through proactive planning.

She summarized with their view that collaboration is the key to cybersecurity and cyber resiliency. That includes collaboration with vendors, clinical staff, and technical staff.


Cyber Resiliency in the Energy Sector Mr. Fred Hintermister, Global Resilience Federation, VP for Energy Protection Mr. Hintermister began by noting that the state of the energy sector is strong. But he noted that, the opportunities for resilience value delivery have never been greater and are climbing rapidly now due to a unique confluence of events

He defined resilience as reducing the magnitude and duration disruptive events and noted that under that definition it is very strong in North America. Indeed, he noted that it has been shown to be 99+ percent over the past six decades. The nature of the regulatory environment looking back has been successful – in addition to good processes and diversity of install we have performance-oriented regulation based on location, needs and resources as well as constraints – all these come into play in talking about future resilience.

He noted that it is changing time for the energy sector. There is greater reliance on cyber, there is the advent of new forms of energy (e.g., solar, wind), and greater decentralization of energy product. He noted that these are sources of both greater risk and opportunity. For example, he noted the need to report on incidents in a different way. The old ways of simply reporting on incidents when something breaks is not sufficiently timely. Rather the focus should be on reporting cyber incidents (even attempted, unsuccessful ones).

He explained how there are various tactical drivers that can influence the strategic energy resilience. As an example, he explained how organization could use the MITRE ATT&CK model (which does collect data at the tactical level) and use it to collect and share information in a way that is understandable by the C-suite. In so doing that would provide information to stakeholders regarding how well they are proactively addressing problems. This related to another point he made, the importance of Focus on processes – how do we report, what do we report, who does the reporting, how do we share the report, how we incorporate the recommendations that result from the report

He had a variety of foundational observations. Refresh cycle has been long and slow. This does not work in today’s world. It is not justifiable in many things – it is now an investment not a cost to upgrade.

He also offered a list of energy sector observables over the past several years and what they could teach us (see figure below).


Mr. Hintermister offered up a few other observations. One is that the concept of IT and OT, cyber and physical, are converging in the energy sector. An attack may arrive via an enterprise/IT vector and leave impact contrails that are effectively operational – bypassing control networks to deliver functional service elimination or degradation. To address this IT and OT elements need to work together. But this will be a challenge as their cultures are not the same. This is an example of a broader point of his – the importance of greater coordination and cooperation between different communities – Sometimes these may involve vendors, sometimes these may involve competitors. But the threat facing these diverse elements does not care about the distinction, and greater “cooperation” is needed to overcome the threats.

Use of Analog Measures to Provide Cyber Resiliency in the Energy Sector Mr. Brad Stephenson, SouthernCo, Senior Manager, Advanced Threats, Intelligence, and Deterrence Mr. Stephenson began with an explanation of what is Southern Company. It is a Fortune 200 company – very large, electric utility in southeastern part of the country. However, it has been diversifying and purchasing electrical and natural gas suppliers across the country.

The organization that he works in, Threat Management and Intelligence, supports the entire company. They consider a variety of factors including:

• Infrastructure and engineering - Management of essential operation components, such as equipment, data, and software

• Cyber threat intelligence - Researches and analyzes trends and technical developments in cybercrime, hacktivism and cyberespionage

• Use case development – Implements Security Event Correlation Rules and Alerts based on Use Cases, and integrates those with other security platforms to aid enhanced visibility, data enrichment and automation for SOC monitoring


• Incident response - Investigates computer related concerns within the organization. Discovers the problem, mitigates the damage, thoroughly investigates the situation while reducing recovery time and costs

• Threat analysis 24/7 – First line of defense. Responsible for monitoring, as well as proactively seeking out cyber security related events

• Fusion Analysis -- Focuses on potential insider threats leveraging advanced User Behavior Analytics

Mr. Stephenson spoke about the various ways that they are focusing on achieving cyber resilience.

• Redundancy: He noted that Southern Company has a large geographic footprint –there is a primary site and a back-up site located elsewhere. This has been in place as a measure of reliability for a long time, but it is also a way of implementing resiliency.

• Telecon: He also noted that they operate their own telecom because they couldn’t get the level of service that they would need in case of a major power event. All their field personnel use it and communication for remote control of substations and some tower to tower communication

• Manual operations: He indicated that there is some use of analog backups for digital controls systems in use at SouthernCo and that there is some talk of legislation requiring this. But there is an associated with this – a large company like Southern can do this but this may not be true of all the smaller suppliers. He also noted that there are some new nuclear reactors being built and they all have digital control systems.

• Insider threat program: He explained that they employ regular personnel, training and policies but then they have enhanced monitoring on those who can sabotage the company in order to continue to deliver power. In addition, they use leading technologies of monitoring both electronic as well as non-digital (like HR) and fusing it all together. Finally. they employ a fusion center with dedicated personnel to centralize and correlate information from sources across the company and monitor activity that crosses risk thresholds based upon information analyzed.

• Informed Preparation: He explained how they interact across sector, government, monitoring the IT environment, department of energy, DHS, FBI, CYBERCOMMAND. This supports their goal of trying to prepare ahead of time and allows them to understand what they can do and how they can help to get restored to normal operations.

• Table-top exercises: He explained that they find these very valuable activities to better understand consequences if things fail.

• Supply diversity: He noted that they have tens of thousands of suppliers – talked about trade-off decisions about risk between going with someone that knows they have problems vs going with someone that doesn’t know they have a problem yet.

Finally, he noted the concern of nation state actors attacking the power grid. He noted how once a nation state has shown they can hack in and it gets publicized then criminals can then hack in more easily.


Panel: Cyber Resiliency in the Financial Sector Mr. Mark Morrison, OCC, SVP & Chief Security Officer / Security Services; Mr. Mark Brubaker, Bank of America, SVP, Cyber Resilience Executive; Mr. Jerry Archer, Sallie Mae, Chief Security Officer The panel began with opening statements by the three panelists.

• Mark Morrison explained that his organization handles options trading. They are moving to cloud using multiple cloud. One of their challenges is how does one justify the amount they are spending and how do you know when you have achieved enough especially in a dynamic threat environment. He also commented about understanding your critical processes and discuss it as a key enabler and not an obstacle.

• Mark Brubaker explained his view that cyber resilience is an aspect of resilience in general. His view is the difference in what we need to do. He noted that a horizontal view is needed across the organization on how things are used to deliver a service (or deliver a mission) this has not been done well (pivot from vertical view). He noted that while cyber is a game changer in how outages happen, but it is about the service not the component that supports the service.

• Jerry Archer noted that his company has transitioned everything to the cloud. The resilience model – they no longer look at it as disaster recover – instead they look at it as part of business continuity. He espoused the benefit of the cloud that they can work from everywhere- we have a multi-cloud solution. He believes that the cloud provides substantial opportunity to do stuff in a more resilient way. Regulators force them to have two instances of cloud, but they are planning to have a third which would be a private cloud.

His view is that resilience is the ability to operate effectively in the face of an ongoing event. A lot of the security capability and stacks is around isolation – hardened against a lateral spread

In response to the question, what are the gaps for achieving greater cyber resiliency in the financial community?

• Mark Morrison provided an example that they have three lease lines supporting their operations. Two are going bankrupt. Critical infrastructure needs to be reassessed – Clouds are now part of that and need to come under federal regulation. The Capital One Cloud breach is sending shock waves through the community – and how organizations are going to go to cloud remains to be seen. He talked about going to commercial clouds but also a private cloud that they are building on their own for some functions.

• Jerry Archer noted that as we move toward the cloud, we will devolve into specialized clouds based on policies. This will go to automation about implementing these policies – commercial offerings try to lock you in to their services. He feels that they are more secure in the cloud than they were before the moved to cloud - they are doing more data collection for data analytics than they were ever before – (can use analytic monitoring also uses tools: SDP, Dome9; in addition can use micro segmentation to limit lateral


spread of malware. They employ 19 layers of protection inbound and 14 outbound. They also use encryption all the time in storage and in transit to protect it from the cloud owners.

• Mark Brubaker discussed that the concept of doing anything from anywhere is great, but it gives the attackers the ability to do the same thing. Visibility and ability to control things has kept them away from cloud. He said that several things need to progress. Tools need to advance – but these are business tools not necessarily a cyber issue. There is a need to identify business resumption plans – what do you need to do – what do you need to do it, how are you going to do it while things are not available; needs a change of viewpoint. H talked about the use of one’s redundant infrastructure that replicates quickly and has the adversary using the replication to take down the entire AD infrastructure – one need to look at one’s business services and how they should work.

In response to the question, how to extend cyber resiliency to the sector level?

• Mark Morrison explained that we need to determine what type of sector wide communications and standards need to be established before bringing up a relationship after an incident has supposedly been contained. Need more sector wide resiliency and recovery requirements.

• Jerry Archer commented that we need rational oversite to know when someone can go back online – need federal regulation.

• Mark Brubaker said there is a concern if any organization in the financial sector goes down, but there is more concern if multiple go down. He agreed that there is need for regulatory oversite.

In response to the question, what would you ask the broader resiliency community to do in the next year or so?

• Mark Morrison posed several questions/challenges for the audience. How do you deal with adynamic environment and agile development? How do you maintain a thing to recover your applications and data and meet your service level agreements? How do we build more resilient applications in the cloud? H noted that third party assessment of security (vendor and non-vendor) needs to be done better. Finally, how do we deal better with open source software and libraries?

• Jerry Archer asked, how do we know we are building the right security? He noted that things are moving too quickly. The only way to be successful in the cloud is if people outside the financial sector are building those tools. The tools we use today all came from other fields – like the defense or academia.

• Mark Brubaker noted that resiliency is not a single industry problem. We need all of us to focus on different aspects to solve the problem. All the services that we rely on also need to be resilient.


NIST SP 800-160 Volume 2, Final Public Draft, Developing Cyber Resilient Systems – A systems Security Engineering Approach Dr. Ron Ross, NIST Fellow

Dr. Ross began by noting that today’s systems are very brittle, and that they rely largely on a one-dimensional strategy of penetration resistance (keeping adversaries from obtaining a foothold). As such, these systems are highly susceptible to devastating cyber-attacks.

He explained how the adversaries are relentless, using concepts such as deception to gain a foothold and then preposition malware, putting them in a position to exfiltrate valuable information or cripple critical missions.

Dr. Ross proposed a new paradigm, that cyber resilient systems should operate more like the human body than like traditional finite state machines. By that he meant that the system should treat malware like the body treats infections; that the defenses are multi-layered, evolve to respond to changing threats, and that the body continues to function (as best it is able) despite the presence of an ongoing infection.

He offered up the concept (from NIST SP 800-160 V2, Final Public Draft) two definitions:

• Cyber Resiliency Engineering: An emerging, specialty systems engineering discipline, applied in conjunction with resilience engineering and systems security engineering to develop survivable, trustworthy systems.

• Cyber Resiliency: The ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by cyber resources.

He explained how cyber resiliency does not exist in a vacuum. Instead it has relationships with other specialty engineering disciplines including reliability, fault tolerance, security, safety, and resilience and survivability.

He explained how reducing susceptibility to the advanced cyber threat requires multidimensional strategy. The first dimension is the traditional approach of hardening the target. The second dimension is reducing the damage to the target from a successful compromise (overcoming the hardening). The third dimension is to make the target resilient.

Dr. Ross then began to talk about the NIST SP 800-160 series. NIST SP 800-160 Volume 1 is focused on system security engineering and discusses 15 life cycle processes. NIST SP 800-160 Volume 2 is focused on cyber resiliency. It is intended to act as a bridge between the system security engineering community and the risk management framework community.


NIST SP 800-160 Volume 2, FPD, lays out the cyber resiliency constructs – goals, objectives, sub-objectives, techniques, approaches, strategic design principles, and structural design principles. He noted that the definition of some of the constructs had evolved in the final public draft of 800-160 Volume 2 from the initial public draft. Dr. Ross noted how the various constructs are interrelated and captured in numerous tables in NIST SP 800-160 V2.

Dr. Ross also explained that the constructs in NIST SP 800-160 V2 were linked to other government efforts. He noted how the document provides a mapping of the 21 objectives in the NSA/CSS Technical Cyber Threat framework to the NIST SP 800-160 V2 cyber resiliency techniques and approaches.

– Each of the 21 NTCTF adversary objectives is mapped against each of the 48 cyber resiliency approaches.

– Illustrates how cyber resiliency techniques and approaches can affect threat events using the NTCTF.

– Mapping identifies which, if any, of 15 effects on the adversary are applicable.

He also explained how the final public draft includes three cyber resiliency use cases – self-driving car, enterprise IT, and a campus microgrid. The use cases provide representative situations in which cyber resiliency is considered by systems security engineering. They show how cyber resiliency concepts and constructs can be interpreted and applied to that situation. Finally, the use cases illustrate how cyber resiliency solutions can be defined or how specific solutions can be applied.

Dr. Ross also noted how the final public draft includes a real-world example – the cyber-attack on the Ukraine power grid a few years ago. For each step of the attack the document identifies potential cyber resiliency techniques and approaches and representative technologies that could potentially be used to mitigate the attacks.

Dr. Ross also noted how many of the elements of the invitational were linked to NIST SP 800-160 V2

• NIST SP 800-160 V2, FPD, was the core of the tutorial • The Ukraine attack and how cyber resiliency could mitigate it was the basis for the table-

top exercise • The AFRL Cyber Survivability Attributes Tool, which was demonstrated at the event,

provides automated support to navigate the various tables in NIST SP 800-160, Volume 2.


Finally, Dr. Ross noted how NIST SP 800-160 V2, FPD, is out for public review and comments through 1 November 2019.

Product Vendor Talks Five of the commercial cyber resiliency vendors who attended the invitational provided a short (5-7 minute) description of their product. The products and presenters were:

Attivo: Don Woodard Illumio: John Westerman Guardicore: Andy Hicks Polyverse: Kiley Williams TripWire: Gabe Authier

In addition, James Reilly of AFRL, gave a short description of their cyber survivability tool.

Panel: Compare and Contrast – Cyber Resiliency Issues Across Domains Milos Manic, Ph.D., Director, VCU Cybersecurity Center Affiliate, Idaho National Laboratory (INL); Mr. Zach Furness, Director of IT Security, INOVA Health Systems; Mr. Bob Bigman, President of 2BSecure

The panel began with the moderator (Richard Graubart of MITRE) explaining the purpose of the panel – to achieve a better understanding of the similarities and differences regarding cyber resiliency across various domains. Each of the three panelists were each then provided time to provide opening statements.

The panelists were then asked what did they see as the greatest threats for which cyber resiliency – rather than simply cyber hygiene – is needed? In other words, what kept them up at night? All three indicated that data breaches were a cause of concern. From the healthcare perspective it was also noted there is a concern (as noted by the earlier medical speaker) of disabling medical devices. Moreover, the panelist noted that there was an additional concern that the attackers could use such devices as a pivot point or changing the data so the doctor would get a false read.

The moderator also asked the panel if there were political, operational, or economic considerations in your sector which might impact selection of cyber resiliency techniques and technologies? Two of the panelists did note that legacy investments might constrain diversity. Also, for some communities, there were potentially political issues with the use of deception.

Since two of the panelists did have experience in the national security community, the moderator inquired if any of those experience could be applied in their current domains. It was agreed that the national security community has a stronger regulatory support system and training for staff to deal with cyber incursions from advanced adversaries.

Finally, the panelists were asked if there was anything that they had heard during the Invitational that would help either their organizations or their sector. Unfortunately, all the panelists had


arrived just before their panel, and thus had not had the opportunity to hear earlier parts of the invitational.

Section 4: Break Out Sessions, Thursday, September 5th Breakout Session: Cyber Resiliency Metrics, Measurements, and Assessments The session was run by Ms. Deb Bodeau of MITRE.

Goal The goal of the session was to share views and work in the area of cyber resiliency metrics, measurements and assessments.

Discussion / Observations MITRE provided a series of slides looking at various aspects of cyber resiliency metrics, measurements and assessments. These slides helped inform discussion, which was very lively. The discussion began with Deb Bodeau of MITRE presenting portions of the MITRE slide deck. She first provided slides trying to differentiate between the concepts of measurement, metrics and assessment (see below).

There was general agreement on the three definitions and their differences. But it was noted at various times during the meeting several members of the audience (and in some cases the MITRE presenters) would occasionally use the incorrect term.

This was followed by a discussion on what makes cyber resiliency metrics hard. There were four elements to this.

1. Scopes vary, and the broader the scope, the harder it is to obtain data … and to ensure its consistent quality

2. System quality properties (of which cyber resiliency is one) are emergent. It is noted in NIST SP 800-160 V1, that” Emergent properties are typically qualitative in nature, are


subjective in their nature and assessment, and require consensus agreement based on evidentiary analysis and reasoning.”

3. Cyber resiliency must deal with advanced cyber adversaries. Such adversaries are by definition, very stealthy and therefore relying on detection of a disruption may be harder.

4. Advanced Adversaries Interact with Systems in Varying and Complex Ways. Traditional resilience metrics tend to assume adversary’s goal is to impact availability. But if the adversary’s goal is to impact confidentiality (e.g., unauthorized data exfiltration) or subtle corruption / integrity attacks, measurement becomes much harder.

Early in the meeting a question was raised from the audience of “what are we attempting to measure?” This led to a comment from one member of the audience that “measuring resiliency may be similar to measuring wetness”. The following comment was that perhaps because cyber resiliency is an emergent property, we might only be able to have resiliency metrics, not measurements.

Deb Schuh of MITRE briefed on mission-centric metrics. One of her key points was that different stakeholders employ/require different metrics. In other words, the metrics that are needed for mission commanders are likely to be very different than those for cyber defenders. The bottom line here is that with regards to cyber resiliency metrics it is all about the mission to determine the appropriate metrics.

One of Deb Schuh’s slides noted (cyber resiliency) Risk = Likelihood x Impact. This led to various interesting discussions. For example, one question was how does one measure likelihood. It was also noted that likelihood is easy to measure for events with a long history such as earthquakes, hurricanes, tornadoes. But cyber resiliency is relatively new, as are the actions of advanced cyber adversaries, and such adversaries make changes to their TTPs in response defensive measures. Therefore, it very difficult to gain a history, and hence a likelihood of such events.

There was also discussion that a determined adversary will eventually achieve system compromise (which is one of those characteristics of cyber resiliency that is somewhat unique), so that perhaps focus should instead be on reducing impact.

This also led to an interesting discussion on the difference between security and resiliency. The premise of the discussion was that security is essentially about hardening, and resiliency is about response after hardening fails (although there was general agreement that there are some techniques and technologies that can be applied in support of both disciplines). It was proposed that not all elements of a system can be made resilient. So perhaps the best approach is to apply security (hardening) to those critical system elements that cannot be made resilient.

Deb Bodeau gave a presentation on two of the cyber resiliency assessment methods and scoring systems that MITRE has developed. The first was MITRE’s Situated Scoring Methodology for Cyber Resiliency (SSM-CR). SSM-CR is a tailorable scoring methodology provides program managers a simple measure of how cyber resilient a given system is, and of whether and how much different alternatives change that measure. SSM-CR, a.k.a. the Measuring the Effectiveness of Cyber Resiliency (MECR) methodology provides priority-weighted assessment


of a system’s cyber resiliency posture – before and after application of potential cyber resiliency improvements.

The second was MITRE’s Adversary Objective Coverage Analysis (AOCA). AOCA builds upon NSA/CSS Technical Cyber Threat Framework (NTCTF). It attempts to answer the question: How well does this [system, solution, requirement or set of requirements, cyber resiliency technique or implementation approach] cover the set of relevant adversary objective; where “cover an objective” means “have a defined effect on one or more representative adversary actions to achieve that objective”. Effects are defined using the structured vocabulary in the FPD of NIST SP 800-160 Vol. 2 which describes 15 possible effects on an adversary (e.g., expunge, preempt, contain, delay, divert).

AOCA provides a mapping of the 48 cyber resiliency approaches described in NIST SP 800-160 Vol 2 against the 21 NTCTF objectives, identifying the possible effects that the approach might have on a given objective. A snapshot of that mapping is listed below.

The MITRE presentations concluded with Rich Graubart presenting on a proposed methodology to evaluate various cyber resiliency assessment methodologies or scoring systems. The evaluation methodology identifies 5 factors to consider (scope, threat, mitigations, scoring, and usability) each of which has various sub-factors. The key point here is that since there is no single best scoring methodology, it is important to understand the relevance of any scoring methodology to a situation.

One of the other attendees offered a short presentation that talked about different solutions for “the age of resiliency”. He proposed three elements to these solutions

• Distributed – distributed services as a solution to DDOS attacks; thus, helping to provide availability

• Immutable – changes that are easier to detect and reverse to prevent unauthorized changes; thus, helping to provide greater integrity

• Ephemeral – make attacker persistence harder to achieve, drive value of assets down to zero; thus, helping to provide greater confidentiality.


His proposal was that rather than try to implement and measure traditional Risk = Likelihood x Impact, it is better to simply try to maximize and measure the Distributed, Immutable, Ephemeral (DIE) triad.

Recommendations/Way Forward The discussion affirmed two points that were made in the plenary presentations, presentations during this track, and prior Invitationals. First, decision-makers want measurements and metrics that could inform their decisions, help them understand their risk posture, and determine the effectiveness of the risk management investments they have made. Second, cyber resiliency metrics have much in common with security metrics and system resilience metrics, sharing some assumptions and challenges (both definitional and evaluative), but are distinctive in their underlying assumption of stealthy and persistent cyber threat actors. Worked examples, showing how cyber resiliency metrics could be defined, evaluated, and used in specific situations would be helpful to organizations seeking to make cyber resiliency metrics part of a larger metrics or risk management regime. The discussion raised two new points, which merit further investigation. First, cyber resiliency metrics are highly situated or contextual; they assume a threat environment, an organizational risk management strategy, and an operational environment. Worked examples could illustrate how the definition, evaluation, and use of cyber resiliency metrics are context-sensitive. Second, while interest is growing in metrics (for risk, cybersecurity, conventional resilience, or cyber resiliency) at a larger scale (e.g., for the financial services sector as a whole), the context-sensitive nature of organizational metrics means that a roll-up from metrics gathered by individual organizations to a sector as a whole would not be meaningful or useful (even if it were achievable). Instead, investigations of larger-scale cyber resiliency metrics should assume externally-based measurements or observations.

Breakout Session: Cyber Resiliency Table-Top This session was developed and led by Mr. Shane Steiger, Principle Cyber Security Engineer, MITRE Corporation.


Goal The purpose was to use a cyber game to study resilience by way of examining the Ukrainian power events of 2015 & 2016.

Discussion / Observations Background The tabletop used two frameworks – the Cyber Resiliency Engineering Framework (CREF) and the Adversarial Tactics Techniques and Common Knowledge (ATT&CK) Framework – the Ukrainian Power Grid events (actual attacks against the Ukrainian power sector) , and a cyber game called MAELSTROM. Mr. Steiger also provided a review of Industrial Controls Systems (ICS). The game used 5 Injects focusing on the control equipment and the devices under control. For this reason, the ICS ATT&CK model was used in this tabletop. This model is meant to address the unique challenges of an automation environment where many of the same techniques used by the adversary in Enterprise ATT&CK are valid against Human Machine Interfaces (HMIs) and Supervisory Controls and Data Acquisition (SCADA) devices in manufacturing plants and power grids. ICS ATT&CK was developed to supplement what became known as Enterprise ATT&CK due to the differing act or objectives or goals of an adversary as well as tactics that are unique to these environments. These environments often span IT environments to Operational Technology (OT) environments.

The participants were able to use cyber resiliency flashcards that provided information on the CREF. A brief review of ATT&CK was provided.

The Scenario There were two attacks on the Ukrainian power grid. One in 2015 and the other in 2016. The events in the first attack (2015) were:

§ On December 23, 2015, a total of 5 energy distribution companies were affected by cyber-attacks on controls devices as part of the Ukrainian power grid.

§ 30 substations were switched off with ~230,000 people left without electricity for a period from 1 to 6 hours.

§ At the same time, a smaller scale cyber-attack was perpetrated on consumers of two other energy distribution companies.

The events in the second attack (2016) were (this attack could be left in a system at one point in time and activated later:

§ On December 16, 2016, Industroyer (also known as Crashoverride) was used as a malware framework considered to launch a cyber-attack on the Ukraine’s power grid.

§ The attack cut a fifth of Kiev, the capital, off power for one hour and is considered to have been a large-scale test.

§ Industroyer/Crashoverride as a malware framework was designed to attack electrical grids.


Using the MITRE ATT&CK for ICS Mr. Steiger showed what actions the adversary took in the two attacks to compromise the power grid. Mr. Steiger then used the Maelstrom card game to facilitate the table-top. Maelstrom is a board game with player cards, threat actor game pieces and ICS tokens with:

§ Attacker deck based on TTPs in ATT&CK™ Enterprise & ATT&CK™ for ICS

§ Defender Deck based on TTPs to disrupt the ATT&CK™ TTPs

§ Progressive Board based on a Cyber Attack Lifecycle (Lockheed Martin’s Kill Chain™)

The game and the table-top consisted of five injects.

Inject 1

The goal was to get participants looking at the cards and using the concept of attack chaining to build out the Ukrainian events of 2015. There was an error in the attacker deck and each team found that error. The question of chaining events came up and the relationships and order of Tactic Category used. It was also noted the length of time the adversary had in the system to study traffic.

Inject 2

The goal was to force the teams to look to other TTPs/attacker cards in the Initial Access Tactic Category if a technique was successfully defended against. The point was to show attacker's resiliency in their campaigns and attack chain choices. Another point was to show that the Initial Access to the Enterprise could grant the adversary's access beyond the Enterprise and into the Operational Technology space. Furthermore, the threat terrain in OT spaces means Initial Access to the IT space through to the OT spaces. The teams were asked which techniques they thought were best. Valid Accounts and Remote Services exploitation were noted as the most effective. However, a supply chain attack was also added as a possibility. This approach did have a higher cost. Nonetheless, the team did note that the more techniques used by an adversary would make them noisier.

Inject 3

The goal was to get participants to look at the Ukrainian 2016 attack as a possible fire and forget cyber campaign with no human needed. They built out the attack chain for the ICS side of the attack with the cards. It was noted that the comparisons of the 2015 events and 2016 events seemed to be less noisy as less ATT&CK ICS TTPs were used. This is because a human was not required for most of the attack once Initial Access was achieved. It was also noted that the Initial Access method although not known could have been an insider or compromised integrator. The dangers of the malware vs. a human required component was discussed.

Inject 4

The goal was to get the teams thinking about tactical, strategic and architectural defenses to the events from 2015 and 2016. The teams were given a hint to start with the NIST 800 160 vol 2 techniques. One team took the hint and started directly at the Techniques level. Their choices


were noted in a picture taken at the TTX below. They then where in the process of translating their choices in the tactical card choices. Other teams wanted to jump straight to the defender cards and not do the analysis from a technique perspective. One team pointed out they wanted to fight at the IT and the OT space to defeat the chains. This was an excellent point, but they did want to tactically step by step approach the chains. Although this is not a wrong answer the techniques analysis would have enabled a broader conversation. Another team took the awesome approach of cost only analysis. They took all the cards and arranged them by cost and effectiveness. They then used the low-cost cards that made sense. This was a very interesting adaption of lessons that was related when playing the game multiple times. However, this approach was very novel and effective in choices. Many of the choices then were checked against the previous year’s approach represented in NIST 800 160v2 Ukrainian use case.

Inject 5

The goal of the last inject was to force the question of impact in terms of destruction, integrity and trust. The teams were meant to think of campaigns as either long disruptions of trust or integrity versus the possible shorter-term impacts of destruction in some context. One team pointed out that there was huge impact from many recent events to brands etc. The question was asked how many of those folks still were in business and the answer was they still are so the impact in that case might not have long term effects. Although, it was later pointed out the destruction with ecological impact like in nuclear scenarios would half a possible radioactive half-life of impact. This point was taken by everyone. Nonetheless, the overall takeaway came back to trust and integrity in this context of ICS impact.


Recommendations/Way Forward The feedback on the table-top was very positive. This appears to be the first instance of a table-top where the threat was based on an ATT&CK analysis of an actual adversary event and the cyber resiliency mitigations drawn from constructs in NIST SP 800-160 V2. There were requests for more such table-tops in future cyber resiliency Invitationals. Moreover, at least four individuals inquired if there was some way MITRE could sell this as a jumpstart service for their own organizations. Break out Session: Deception This session was led by Dr. Stan Barr and Ms. Mary Yang, both of MITRE.

Goal The goal of the break out session is to have a robust discussion around cyber deterrence and adversary engagement – if organizations are thinking about their cybersecurity strategy from an adversary engagement perspective; why or why not – with the understanding that a core component of cyber deterrence and adversary engagement relies on deception technology.

Discussion / Observations The meeting began with introductions around the room. Each person was asked to state their name, organizational affiliation, and why they chose the deception breakout session and what they hope to gain from the session.

The organizers asked a series of questions that generated a variety of feedback from the audience. Several of the questions and responses are listed below (no attribution is provided intentionally).

Question: How many people in the session were using deception (question by the session organizers)? Approximately half of the room indicated that they were.

Question: How does deception fit into your organization’s overall cybersecurity strategy? A representative from DoD indicated that deception was being looked at as a strategy to deter or deceive adversaries to buy time. Deception somewhat falls between the cracks. Unlike network security there is now government mandate to employ deception. This in turn impacts funding.

Question: What barriers or challenges would have to be overcome to leverage deception for cyber deterrence and adversary engagement? There were a variety of responses, some were technical, some were non-technical.

• Among the more technically oriented responses were: o Complexity and knowing what provides the best bang for the buck o Not knowing the adversary tradecraft level so organizations did not know what

type of deception to use o Others noted that it is very difficult to execute a controlled experiment using an

adversary persona; it is hard to define one for red teams. Related to that was a comment that it is hard to characterize the ROEs; hard to make it realistic


because if you think about your adversary, they are likely using social media so they will need to be monitored as well

o In applying deception, one will encounter multiple false positives o Real networks are noisy, and that impacts effectiveness of deception

• Among the non-technical oriented responses:

o Decision makers did not currently understand the role and need for deception o There are multiple legalities regarding the use of deception: need to consult with

lawyers; there are huge gaps in the US law regarding deception o It can be hard to measure the effectiveness of using deception, that in turn makes

it difficult to justify the investment o There is an operational impact to deception -- one has to make sure that the “good

guys are not the ones being deceived. Many are not ready to introduce deception because of this

o Someone noted that one cannot discuss deception without giving away the fact that deception was being used

Question: What do you think would change minds to use deception? Again, this solicited a variety of responses:

o Deception needs to be realistic, so it is believable. o Approach deception from a crawl, walk, run perspective and consider what you

are attacking/protecting. Related to this was the comment that don’t let perfect hold you back from deploying deception and that weak deception succeeds more often than not.

Question: What goals would you want deception to help you meet?

o One response was the hope that deception might help provide insight into what was happening on one’s network.

o Related to this was the hope that deception might provide the defender first-hand experience as to what is happening within the network and provide that analysis with a small staff.

o Another response was that hopefully deception might change the dynamics of the game – that the hunters becomes the hunted.

Recommendations/Way Forward The deception track was well received. One suggestion that came out of it was that due to the sensitive nature of the use of deception (and that some organizations are understandably uncomfortable discussing the topic in detail in a public forum) that MITRE help put together follow-up deception technical exchange meeting (TEM) at a classified level.


Break out Session: Cyber Resiliency in Weapon Systems The session was led by Mr. John Garstka, Director, Cyber, Office of the Chief Information Security Officer, Office of the Undersecretary of Defense for Acquisition and Sustainment. This breakout session differed from the others in that it was limited to government employees, their FRRDCs, National Labs, or contractors. Some of the material presented and possibly used in discussion may have been sensitive but unclassified. As such, only a very preliminary description of this track is provided here. Those interested in a more detail description of the track are encouraged to contact Rosalie McQuaid ([email protected]) of the MITRE Corporation.

Goal The goals of the weapon system track were four-fold:

• Examine challenges and approaches of applying cyber resiliency to weapon systems • Introduce components of a draft resiliency framework for weapon systems; foundational

principles and a self-assessment questionnaire • Determine coverage/gaps, refinements and usability of framework components • Provide a forum for ongoing discussions in this area

Discussion / Observations After introductions an overview of the Cyber resiliency Technical Directive Agent (TDA) was provided. The initial goal of the TDA was to produce a cyber resiliency framework for evaluating existing and new weapon systems. In the initial stage of the breakout session attendees were provided with the details of the framework (which cannot be shared in this document).

At a high level the framework consists of:

1. A baseline assessment of the weapon system 2. A gap assessment of what else needs to be done 3. An analysis of alternatives – what will give the organization the best value

In support of the baseline assessment, the framework provides a set of 23 cyber resiliency foundational principles, that are derived from the constructs of NIST SP 800-160 Vol 2 FPD. These, in turn, each have one or more supplemental principles. The 23 foundational principles are divided across four levels:

• Mission (4 principles): The strategically desired organizational goal; what the organization is trying to accomplish or achieve; what the operators are using the set of related weapon systems to achieve

• Operational (5 principles): The weapon system or systems that are employed to accomplish a mission, explicitly including people and processes as part of the system

• Programmatic (8 principles): A funded effort that provides a new, improved, or continuing materiel, weapon or information system, or service capability in response to


an approved need; funded effort responsible for the development, update and sustainment of weapon systems

• Technical (6 principles): The technologies, systems, and processes produced by a program and that underlie operational components.

To enable organizations to assess how well their weapons systems are doing relative to the foundational principles, a questionnaire had been developed and was shared with the session attendees.

In the second stage of this breakout session, a notional aircraft weapon system was introduced. The weapons system was totally unclassified and fictional, bearing no linkage to any real system.

The third stage involved the attendees breaking up into 5 groups. Each group was directed to answer different parts of the questionnaire. They were to use the questionnaire to determine the notional weapon system baseline and gaps. To support their efforts each group was provided

– Conference paper containing framework overview – Questionnaire with notional weapon system mission questions filled out – Foundational principles paper (several per group)

An intended outcome of the exercise is an assessment of the 1) comprehensiveness, 2) usability, and 3) effectiveness of the questionnaire and the framework.

Issues, Challenges or Outcomes The detailed outcomes of the five groups cannot be discussed in this document for reasons cited earlier. But there were some high-level general findings that occurred in many of the groups.

• Suggested changes: o The terms employed in the questionnaire need to be better defined o The questionnaire could benefit from being simplified. Perhaps with more yes/no

questions o Many of the questions are too vague, and high level. Greater specificity is needed,

and care should be taken that the language of the questions should be consistent with that of the level (e.g., use more acquisition language for programmatic level questions).

• Good points about questionnaire: o Questionnaire forces program office to talk about resilience, which in and of itself

is a positive and not necessarily done today o Program managers may realize that they have some amount of mission resilience,

even if unintentional o The way the levels were broken up (mission, technical, program, operational) was

well received.

9th annual secure and resilient cyber architectures ... · renamed “invitational” and held in...

Documents