8th annual secure and resilient cyber architectures

1 ©2019 The MITRE Corporation. All Rights Reserved Approved for Public Release; Distribution Unlimited. Case # PR 19-02172-5

8th Annual Secure and Resilient Cyber Architectures Invitational & Training Event: 2018 Proceedings


Table of Contents Overview ........................................................................................................................................ 4

Background .................................................................................................................................... 4

Prior Years: 2010 - 2017 ............................................................................................................. 4

2018 .......................................................................................................................................... 5

Introduction ................................................................................................................................... 6

Section 1: Tutorial, Monday, May 7th ......................................................................................... 8

Section 2: Presentations, Tuesday, May 8th ................................................................................ 9

Kickoff Presentation .................................................................................................................... 9

Building Cyber Resilient Systems: A National and Economic Security Imperative .................. 9

Cyber Resiliency Without Detection ......................................................................................... 10

Micro-Virtualization .................................................................................................................. 11

Operationalizing Resiliency Track ............................................................................................ 12

Consequence-driven Cyber-Informed Engineering (CCE) Methodology ................................ 12

Operationalizing Resiliency: An Infrastructure Lifelines Perspective ...................................... 13

Large-Program Cyber Resiliency .............................................................................................. 14

Section 3: Presentations Wednesday May 9th ........................................................................... 15

Resilience in Elections .............................................................................................................. 15

Cyber Resilience and Response Impressions from the Field .................................................... 16

Cyber Resiliency for Weapons Systems ................................................................................... 17

DoD Weapon System Engineering for Operations in Contested Cyberspace Environments ... 18

Cyber Resiliency for Weapon Systems ..................................................................................... 19

Applying Cyber Prep 2.0 and Cyber Resiliency to Build Out a Risk Universe ........................ 20

Product Vendor Talks .................................................................................................................... 21

Section 4: Break Out Sessions, Tuesday, May 8th .................................................................... 21

Breakout Session: NIST SP 800-160 Volume 2 ....................................................................... 21

Goal .................................................................................................................................. 21

Discussion / Observations ..................................................................................................... 22

Challenges ............................................................................................................................. 23

Recommendations/Way Forward .......................................................................................... 24

Breakout Session: Resiliency without Detection - Mini-Table-Top Exercise (TTX) ............... 24

Phase 1, 45 minutes ............................................................................................................... 25


Phase 2, 45 minutes ............................................................................................................... 26

Phase 3, 45 minutes ............................................................................................................... 27

Readout, 30 minutes .............................................................................................................. 27

Break out Session: Operationalizing Cyber Resiliency ............................................................ 28

Goal .................................................................................................................................. 28


Issues and Challenges ............................................................................................................ 29

Section 5: Break Out Sessions, Wednesday, May 9th ............................................................... 30

Using Cyber Resiliency to Mitigate Adversary Actions ........................................................... 30

Goal .................................................................................................................................. 30


ATT&CK and CREF Discussion .......................................................................................... 31

ACR Discussion .................................................................................................................... 32

Challenges ............................................................................................................................. 33

ATT&CK and CREF Challenges ...................................................................................... 33

ACR Challenges ................................................................................................................... 33

Recommendations/Way Forward .......................................................................................... 33

Cyber Resiliency in Weapons Systems ..................................................................................... 34

Goal ........................................................................................................................................ 34

Observations .......................................................................................................................... 34

Challenges ............................................................................................................................. 36

Way Forward ......................................................................................................................... 36


Overview May 2018 marked the eight year in which approximately 150 subject matter experts (SMEs) in cyber resiliency from government, industry, and academia came together in McLean, VA, for collective work on topics of common policy and engineering concern. For two days, the 8th Annual Secure and Resilient Cyber Architectures Invitational & Training Event accelerated recognition and adoption of cyber resiliency with a focus on organizations.

Background Prior Years: 2010 - 2017 The first workshop, held in October 2010, established the initial community and shared architectural, technical, and policy perspectives on cyber resiliency. The second workshop, held in May 2012, focused on collaborating to develop a communal view of resiliency frameworks, engineering principles, and metrics [1]. The third workshop, held in June 2013, centered on identifying favorable conditions for use of specific resiliency techniques, assessing the use of techniques in enterprise architectures, and developing use cases [2]. The fourth meeting, now renamed “Invitational” and held in May 2014, emphasized applying cyber resiliency to space-based systems and critical infrastructure, designing a cyber resiliency challenge, and identifying roles played by cyber resiliency throughout the systems engineering life cycle [3]. The Fifth Annual Secure and Resilient Cyber Architectures Invitational, held in May 2015, concentrated on taking stock of the state of cyber resiliency: the lessons learned and the remaining challenges to overcome. It sought community consensus on the theme of Cyber Resilience: Looking Backward (What Has Worked? What Has Not?), Looking Forward (What New Challenges Must Be Faced?). Keynote speakers included representatives from the National Institute of Standards and Technology (NIST), US Navy, Indiana University, and Bit9 + Carbon Black [4]. The Sixth Annual Secure and Resilient Cyber Architectures Invitational, which took place on 18–19 May 2016, centered on the theme of Institutionalizing Cyber Resiliency [5]. Four keynote speakers were followed by panel discussions inclusive of industry leaders. Three working groups furthered knowledge sharing by focusing on: cyber resiliency and system security engineering cyber resiliency and an organization’s cybersecurity program, and cyber resiliency and acquisition. In addition, vendor booths and representatives displayed leading-edge cyber resiliency offerings.

The Seventh Annual Secure and Resilient Cyber Architectures Invitational and Training Event was held May 9-10th, with an optional tutorial held the afternoon of the May 8th. The addition of the tutorial was the reason that that “Training Event” was added to the title. The event included four presentations, a panel, four facilitated working groups, and selected vendors booths and presentations by those vendors. Topics addressed included: cyber resiliency in the financial community, cyber resiliency and architectures, measuring the effectivenss of cyber resiliency, and cyber resilience in weapon systems.


2018 The rest of this report focuses on the 8th Annual Secure and Resilient Cyber Architectures Invitational and Training Event. These proceedings present a summary of the keynote talks, the panel discussion, and working group tracks. The Cyber Resiliency Invitational Committee believes the Invitational serves a larger mission: to advance the field of cyber resiliency for our sponsors and nation. Additional materials from the invitational and briefings can be found at http://www.mitre.org/cyberworkshop. The committee welcomes comments from readers through the contact email address: [email protected].

The Cyber Resiliency Invitational Committee August 2019


Introduction The 8th Annual Secure and Resilient Cyber Architectures Invitational & Training Event included thirteen presentations, a lightning session, five three-hour breakout sessions, and ten selected vendors presented over two days. In addition, there was an optional tutorial the afternoon prior to the commencement of the event which is discussed briefly in Section 1.

Section 2 summarizes the following seven presentations provided on Tuesday May 8th.

• Building Cyber Resilient Systems: A National and Economic Security Imperative, by Dr. Ron Ross, NIST Fellow

• Cyber Resiliency Without Detection, by Dr. Vipin Swarup, MITRE Corporation • Micro-Virtualization, by Dr Ian Pratt, Bromium • Operationalizing Resiliency Track, by Ms. Emily Frye, MITRE Corporation • Consequence-driven Cyber-Informed Engineering (CCE) Methodology -- Engineering

out the cyber risk from things that must not fail, by Mr. Andy Bochman – Idaho National Labs

• Operationalizing resiliency: An infrastructure lifelines perspective, by Dr. Elise Miller-Hooks, Hazel Chair in Civil Engineering, George Mason University

• Large Program Cyber Resiliency, by Mr. Skip Reindollar, MITRE Corporation

Section 3 summarizes the following six presentations and the lightning session provided on Wednesday May 9th.

• Resilience in Elections, by Mr. Jeremy Epstein, National Science Foundation • Cyber Resilience and Response Impressions from the field, by Mr. Peter Mitchener, FBI

Senior National Intelligence Officer for Cyber • Cyber Resiliency for Weapon Systems, Mr. Daniel Holtzman, United States Air Force • DoD Weapon System Engineering for Operations in Contested Cyberspace

Environments, by Ms. Melinda Reed, Deputy Director for Program Protection Assistant Secretary of Defense, Research and Engineering (ASD(R&E))

• Cyber Resiliency for Weapon Systems, by Col. Ed Masterson, Acting Director, Cyber Resiliency Office for Weapon Systems, USAF

• Applying Cyber Prep 2.0 and Cyber Resiliency to build out a Risk Universe, by Mr. James Mailliard, Vice President Cyber Security Governance, Risk and Compliance, Elsevier

• Vendor Product Lightning Round Talks

Section 4 summarizes the three breakout sessions provided during the afternoon of the May 8th.

• NIST SP 800-160 Volume 2 – Ask the Authors, overseen by Dr. Ron Ross of NIST, Deb Bodeau of MITRE, Rich Graubart of MITRE.

The initial public draft of NIST SP 800-160 Volume 2 – Systems Security Engineering: Cyber Resiliency Considerations for the Engineering of Trustworthy Secure Systems was released for


review on 21 March. The purpose of this breakout session was to provide reviewers the opportunity to offer comments informally and to ask the authors questions about the document.

• Cyber Resiliency Without Detection Table-Top o This breakout session built upon and extended the morning briefings by Vipin

Swarup and Ian Pratt. Its purpose was to demonstrate the viability of cyber resiliency without detection though a series of table-top injects.

• Operationalizing Cyber Resilience o The breakout session built upon and extended the morning briefings. It provided

the attendees the opportunity to share experiences and challenges that they have encountered in their attempts to operationalize cyber resilience. And to discuss unmet needs they see or predict, while turning resilience guidelines and frameworks into practice.

Section 5 summarizes the two breakout sessions provided during the afternoon of the May 9th.

• Using Cyber Resiliency to Mitigate Adversary Actions o Analysis of adversary actions and cyber resiliency are often considered two

distinct concepts. This track challenged this belief by examining how adversary focused frameworks, in conjunction with cyber resiliency solutions, could help to counter advanced cyber-attacks.

• Cyber Resiliency in Weapon’s Systems o This breakout session built upon the keynote presentations of the morning. It

provided an opportunity for attendees to discuss challenges in making weapon systems resilient as well possible ways to bring to bear policy, technology, procedures and expertise to enhance the resiliency of such systems.

In parallel to the presentations and breakout sessions, during both days there were vendor booths presenting cyber resiliency enabling products from the following vendors: Akamai, Attivo Networks, Bromium, Cryptonite, Illumio. Illusive Networks, Javelin Networks, Morphisec, Polyverse, and Symantec.


Section 1: Tutorial, Monday, May 7th

A tutorial on cyber resiliency was presented by Ms. Deb Bodeau, Mr. Rich Graubart, and Ms. Rosalie McQuaid, all from the MITRE Corporation. An outline of the tutorial is presented below.

The presenters are also the authors of the initial public draft of NIST SP 800-160, Volume 2 -- Systems Security Engineering: Cyber Resiliency Engineering Considerations for the Engineering of Trustworthy Secure Systems. They drew from that document for much of the tutorial. Discussions covered:

• Definition of cyber resiliency • Why cyber resiliency is needed • Explanation (and definitions) of the various cyber resiliency constructs (goals, objectives,

techniques, approaches, design principles) • Effects cyber resiliency have on adversaries • Linkages between Volume 1 and 2 of NIST SP 800-160 • Linkage between cyber resiliency and NIST SP 800-53

In addition, the tutorial covered material not in NIST SP 800-160 Volume 2, such as discussions on cyber metrics and measures of effectiveness and means of assessing the resiliency of a system.

The tutorial was intended to provide a common basis and understanding of cyber resiliency prior to the commencement of the Invitational.


Section 2: Presentations, Tuesday, May 8th Kickoff Presentation Dr. Vipin Swarup, Director, Cyber-Resilient Systems, The MITRE Corp.

Dr. Swarup, MITRE, served as the master of ceremonies for the Invitational. He provided an overview and look back. The purpose of the event, as stated by Dr. Swarup, was:

• To bring the community together to work collectively to accelerate recognition and adoption of Cyber Security Resilience, and

• To recognize that achieving success in cyber resiliency is beyond any one organization or sector, and success in cyber resiliency requires collaboration across government, industry, academia, and FFRDCs.

Dr. Swarup reviewed the varied presenters of the Invitational for the past years (listing those from government, industry, academia, and FFRDCs/National Labs) and a concise summary of the progression of cyber resiliency over the past seven annual events as well as the impact of the Invitational.

Building Cyber Resilient Systems: A National and Economic Security Imperative Dr. Ron Ross, NIST Fellow

The focus of Dr. Ross’s talk was “NIST SP 800-160 Volume 2: System Security Engineering – Cyber Resiliency Considerations for the Engineering of Trustworthy Systems.” However, prior to discussing the publication, Dr. Ross set the stage for its need.

Dr. Ross explained how cyber resiliency represents the failure to produce the trusted systems we thought we could produce over 40 years ago. He explained how technology overwhelmed us, that our appetite for advanced technology continues to overwhelm our ability to protect it, and that the complexity of today’s systems is our number one threat. Dr. Ross proposed a new paradigm – look at the problem in three dimensions:

• Harden the target (best practices and processes) • Limit damage to the target – some attacks are still going to occur. This is the

necessary part (not the sufficient part) of CR – either in time or space – limit how long the adversary is there or how far the adversary can spread.

• Make the target more resilient – this should be done from the system’s point of view, not just the cyber perspective. This last bullet led into the discussion of NIST SP 800-160 Volume 2.

Dr. Ross noted how much of the framework provided in NIST SP 800-160 Volume 2 came from previous work by MITRE. He noted that while there are multiple frameworks available, none are


perfect. The approach was to create/select the best one, make necessary changes, and stick with it. Dr. Ross continued by providing a high-level description of NIST SP 800-160 Volume 2. Highlights included:

• Cyber resiliency is the ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by cyber resources.

• The publication is focused on mission or business of an organization with a critical assets’ focus.

• The publication focuses on the presence of Advanced persistent threats (APT). APTs are stealthy in nature, persistent, and have the ability to adapt to defenders’ actions. The reason for this focus is that APTs are capable of successfully going after high value assets.

• The publication assumes the adversary will be successful in compromising a system. It is not realistic to assume that the APT can be kept out of a system or be quickly detected and removed from that system, despite the quality of the system design, functional effectiveness of the security components, and trustworthiness of the selected components.

• The publication also assumes the adversary will maintain a presence. This means that even if the system appears to be operating as it should, it does not mean the adversary is not already in the system. Adversaries can, and do, pull triggers whenever they want.

Dr. Ross noted the importance of looking at cyber resiliency in conjunction with other disciplines: reliability, fault tolerance, security, resilience and survivability, and safety. He listed the various constructs that comprise the cyber resiliency engineering framework, its goals, objectives, techniques, approaches, design principles, and showed how they were related.

Cyber Resiliency Without Detection Dr. Vipin Swarup, Director, Cyber-Resilient Systems, The MITRE Corp.

Dr. Swarup began his remarks with stating that ten years ago, there was the realization that state and nonstate actors were breaking into systems -- and were staying in those systems. As a result, the concept of hardening systems to prevent the adversary from gaining a foothold was no longer realistic as the sole approach to dealing with APTs. This gave rise to threat-based defense. This defense largely began as a “detect and response” activity. The problem with this approach, however, was that by the time that the adversary was detected (on average 100 days after the adversary had penetrated the system), harm had already been inflicted on critical missions. Therefore, what became apparent was the need to deal with the adversary prior to detection (also known as left of detection), with a zero-detection architecture.

Dr. Swarup provided a short overview of the various cyber resiliency constructs of the MITRE and NIST SP 800-160 Volume 2 Cyber Resiliency Engineering Framework (CREF). Various


technologies (segmentation, client micro-segmentation, service micro-segmentation, zero trust architectures) were presented next, each of which could help achieve/support resiliency without detection architectures.

Dr. Swarup provided an overview of one of the afternoon’s breakout sessions including a table-top exercise (TTX) designed to show how cyber resiliency without detection could work.

Micro-Virtualization Dr. Ian Pratt, President, Bromium, Inc.

Dr. Pratt opened by stating a fundamental problem in cybersecurity is a lack of isolation. He noted how the root of the problem likely began with multiuser computers. Those systems were designed at a time before the concepts of web browsers and propagation of software existed. There could have been no assumption of going out to the Internet to download pieces of code.

Dr. Pratt asserted he does not believe machine learning will help systems become more cyber resilient because of the common belief that defenders have to be successful all the time whereas attackers only need to be successful once. In addition, attackers may employ machine learning too to maintain their ability to morph and persist.

Dr. Pratt proposed a different course of action where separate environments are created for each task. Calling the approach “micro-segmentation”, he explained how it could be used to provide multiple resiliency capabilities:

• To reduce a system’s attack surface • To break the persistence of an APT • To provide a practical means of implementing the principle of least privilege • To deceive the attacker • To assist in forensic data collection • To protect high-value applications from malware, etc.

Micro-segmentation can be used to isolate mutually untrustworthy tasks via a micro visor (micro hypervisor). “Systems” can be looked at as hardware, kernel, operating system libraries/utilities, and applications. Much of the code was developed in the 1980’s without security in mind. Dr. Pratt went on to explain how micro-segmentation could be used to safely browse the Internet with different virtual machines (VMs) being created for each website a user browses. Furthermore, the VM’s would not connect to an organization’s intranet. To preserve the user experience, relevant cookies could be extracted when the VM is killed, and reinserted when the a new VM is created to re-access that particular website. In addition, micro-segmentation can provide protection for high value applications and for applications employed in environments of dubious trust (e.g., BYOD scenarios).


Operationalizing Resiliency Track Ms. Emily Frye, Portfolio Director, National Protection & Response, The MITRE Corp.

Ms. Frye served as both organizer and lead speaker for the track. The other three speakers were Mr. Andy Bochman, Dr. Elise Miller- Hooks, and Mr. Skip Reindollar. Ms. Frye opened the track by discussing, Crossing the Chasm by Jeffry Moore. A basic tenet of the book was there are people who are early adopters (visionaries) of technologies, and there are people who are pragmatists. Transitioning technologies and concepts from early adopters to pragmatists is essential if the technologies/concepts are to propagate and succeed. She stated the same is true of cyber resiliency. Ms. Frye concluded her opening remarks by introducing the other speakers. Consequence-driven Cyber-Informed Engineering (CCE) Methodology Mr. Andy Bochman, Senior Grid Strategist, Idaho National Labs

Mr. Bochman began with a quote from Richard Danzig: “Successful strategies must proceed from the premise that cyberspace is continuously contested territory in which we can control memory and operating capabilities some of the time but cannot be assured of complete control all of the time or even of any control at any particular time”. He continued the discussion, noting from Dan Geer, that the more we build and deploy these [cyber] systems, the greater is our dependency on them with associated higher the risk. He went on to quote Michael Assante “Cyber hygiene is helpful for warding off online ankle biters and if done perfectly in a utopian world, might thwart 95% of attackers.” In the real world, however, “It registers as barely a speed bump for sophisticated attackers aiming at a particular target.” Mr. Bochman cited the SANS Top 20 as an example of cyber hygiene. Mr. Bochman proceeded to note that an appropriately skilled and resourced attacker will succeed, and organizations will be compromised. To deal with such adversaries and compromises he proposed the Consequence-driven Cyber-Informed Engineering (CCE), four step process:

• Consequence Prioritization – Determine critical functions and high-consequence events; identify what cannot fail through ruthless prioritization based on consequences

• System of system analysis – Examine how the critical function is achieved; identify the key information access and actions an attacker must take to produce an effect

• Consequence basted targeting – illuminate where the control system is vulnerable by thinking like an attacker (networks, supply chain, close access attacks)

• Mitigation and protections – engineer out the cyber risk; interrupt the attacker’s progress with simple and complex engineering controls

Mr. Bochman continued by emphasizing that because of the determination and persistence of the adversary, organizations need to be ruthless in their prioritizations, since everything cannot be preserved/safeguarded.


In the interim, organizations are advised to:

• Keep performing hygiene as well as they can • Ruthlessly prioritize and determine what needs different types of protection • Mitigate risk and determine what can be done if attacks are successful • Change the culture. Bring engineering into security/cyber in a similar way as safety

engineering has been integrated

Operationalizing Resiliency: An Infrastructure Lifelines Perspective Dr. Elise Miller-Hooks, Hazel Chair in Civil Engineering, George Mason University

Dr. Miller-Hooks began by explaining that cyber is one of 18 infrastructures. It is important itself and because its support of other key infrastructures, e.g., transportation, supply chain, communications, and water/wastewater.

There are multiple classes of hazards an organization may need to deal with: natural hazards, malicious attacks, technical /accidental hazards. It is important for an organization to be resilient to multi-hazards. Stakeholders and organizations need to be aware that if they become resilient to one hazard, it may make the organization more vulnerable to others.

Dr. Miller-Hooks explained adversaries may be waiting for a natural hazard to occur, e.g., a hurricane in a certain geographical location, to exploit an organization’s and system’s vulnerabilities. Such a time provides an opportunity for adversaries to launch zero-day attacks, for example. The original hazard serves as cover for an adversary by distracting a victim organization. In addition, the victim organization may take actions during a natural hazard which leaves itself more vulnerable. Advanced adversaries are very patient in waiting for such situations to happen.

Additionally, Dr. Miller-Hooks addressed the role that modeling plays for various infrastructures, she explained different research methods for dealing with the concerns of classes of infrastructures. What is modeled and what is considered critical for resiliency will vary across infrastructures. For example, in the case of ports, an organization may model port traffic, both incoming and outgoing. In the case of airports, visibility, runways, and runway usage may be modeled. For ground, public transit systems, socioeconomic characteristics come into play and should be modeled. Every person using a public transit system has varying capacities to be flexible in accepting alternates. Dr. Miller-Brooks concluded by summarizing her key points.


Large-Program Cyber Resiliency Mr. Skip Reindollar, Principal Software Systems Engineer, The MITRE Corp.

Mr. Reindollar reviewed the use of cyber resiliency support functions for a large-scale civilian agency program. The agency has a complex, enterprise IT, system, with cross agency stakeholders and containing sensitive personally identifiable information (PII) As part of MITRE’s effort, a number of methodologies were surveyed for applicability including MITRE’s Crown Jewel Analysis (CJA) and MITRE’s Threat Assessment Remediation Analysis (TARA). The team took CJA and TARA results and linked them to the agency’s system engineering lifecycle (SELC). The results became sponsor and/or contractor actions during development, implementation, and sustainment

CJA provided a mission impact analysis. In so doing, it allowed the team to identify dependencies among:

• Mission goals • Operational tasks • System functions • Cyber assets

The team worked with sponsor business team to validate the CJA results and to identify critical cyber assets.

The team used TARA to provide both Cyber Threat Susceptibility and Cyber Threat Remediation analysis. TARA allowed the team to organize resiliency needs according to threat categories:

§ denial of service or degradation of service § data exfiltration § data manipulation § external pivot § insider threat § cloud computing considerations TARA also allowed for: prioritizing threats based on likelihoods and impacts to system,

assessing susceptibility/vulnerability of critical cyber assets to the threats, and an overall refined focus on critical cyber assets. A chief outcome was the identification of 87 mitigations.

Additional outcomes of the effort were: the identification of specific actions for program and project leaders, results linked directly to executable SELC tasks according to SELC phases, and the integration of actions into the program schedule and/or risk management plan.


Section 3: Presentations Wednesday May 9th Resilience in Elections Mr. Jeremy Epstein, Deputy Division Director, Computer and Network Systems, National Science Foundation (NSF)

Mr. Epstein started by expressing his view that we need to acknowledge elections are fundamentally cyber-enabled human systems. As such, there are numerous and varied steps in running elections, each with its own set of diverse threats and countermeasures.

Mr. Epstein stepped through several examples to illustrate. First, a threat model for voter registration would need to include: 1) discarded forms at registration locales, such as, post offices, election offices, and 3rd party registration offices, 2) unauthorized changes of address or party affiliation, 3) the registration of non-existing citizens, or, of existing citizens without approval.

Next, Mr. Epstein discussed threats arising from in-person voting via unauthorized voters and duplicate votes. While in-person voter fraud is considered rare according to Mr. Epstein, several cases have been recently documented in Virginia, causing national concern. Countermeasures may result in their own issues. While day-of-voting signatures may be checked against original signatures, the measure may be weak because of natural changes in human signatures. Voter identification, such as drivers’ licenses and passports, generates socioeconomic and immigration controversies.

The primary threat posed by absentee voting arises from unauthorized people requesting or returning ballots. Mr. Epstein stated that there are very few countermeasures available, except for signature verifications as previously described.

Internet voting has gained increased interest over the past few years. It is allowed in 34 states primarily for deployed, military personnel. The general public has often questioned why Internet voting is not allowed in more states by more categories of voters, citing on-line banking as an analogous example. Mr. Epstein drew strong comparisons when reminding the audience of numerous verifications performed by banks before customers execute one on-line transaction. In addition, Mr. Epstein asserted that banks have the opportunity to perform accounting corrections, should fraud occur. Elections, however, do not have such an opportunity, given the immediacy demanded for election results, especially during national elections.

As a final example, Mr. Epstein discussed ballot counting. He asserted the underlying challenge is to ensure accountability balanced with the need to preserve the secrecy of a ballot. The main threat challenge is the ensuring of a chain of custody of cast ballots.

In closing, Mr. Epstein stated resiliency is already a core element of elections and, that despite editorials to the contrary, elections in the US are overwhelmingly executed in a fair manner.


Cyber Resilience and Response Impressions from the Field Mr. Peter Mitchener, Senior National Intelligence Officer for Cyber, Federal Bureau of Investigation (FBI)

The focus of Mr. Mitchener’s talk was the establishment of a public-private partnership sponsored by the Department of Homeland Security (DHS). The partnership team was comprised of representatives from CIA, DHS, FBI, MITRE, various state agencies, and some private sector members, e.g., academia, commercial facilities, energy, and financial services.

Each year, approximately eight security issues are reviewed by the partnership team and in 2018 a key topic of interest was cyber resiliency and response. Specifically, the partnership explored how private and public sectors handle responses to adversarial activities.

The research questions of concern for the team were:

• How are private sector organizations addressing cyber resilience, particularly regarding how to withstand adversarial activity? How do these approaches compare to similar government efforts?

• What are the most common challenges to implementing effective resilience strategies for both government and private organizations?

• Are there best practices that may be widely applied, and if so, what are they? • What is the state of public-private collaboration on resilience? Are there areas for

growth and improvement?

Mr. Mitchener highlighted challenges and opportunities agreed upon by the partnership. The challenges include the following.

• Technology refresh cycles for large, government systems are too long, causing a real lag in addressing knowledge of adversaries’ technologies. For example, on average, aircraft have a 25-year refresh cycles. There is also a challenge to shorten such cycles and to determine who is, or should be, doing the refreshes. Supply chain issues are connected to this challenge as well.

• There are limitations imposed on government systems and their ability to become more resilient because of inherent dependencies on industry vendors. First, the number of vendors in the large systems space is small, and second, their incentives to become more security conscience may be limited by very narrow profit margins, making such improvements not commercially viable.

• The number of available and properly skilled people to staff a robust cyber workforce is limited. For example, the partnership believes the workforce as a whole should have an understanding of threats, attack vectors, and countermeasures. Yet, the workforce is lacking in this area.


• There are legal and regulatory liability concerns regarding the sharing of information and technologies. There is also confusion about what can be shared and with whom. For example, if a large US network experiences a cyber event or breach, the federal government does not own its infrastructure. This is a barrier for the government to directly access the network for threat diagnosis and the build of future resiliency against such threats.

• Certain commercial organizations, if victims of a breach, are reluctant to work with the FBI’s investigation of it for fear of government discovery of company non-compliance issues.

The second half of Mr. Epstein’s talk addressed opportunities identified by the joint partnership.

• There is a large movement towards stronger user authentication approaches like biometrics and two-factor authentication. The goal is to reduce the risks posed by social engineering and attacks.

• Discussions between CEOs and CISOs are becoming more commonplace, thus reducing the disconnects in understanding threat concerns. There are, of course, challenges when defining resilience for certain organizations both in boardrooms and across industries.

• The legal landscape is influenced by image and reputation of an organization and CEOs are especially worried about anything that may tarnish them. Thus, organizations are more open to seeking out red teaming for their systems. In closing, Mr. Epstein presented a balanced perspective of challenges and opportunities going forward. Partnerships between public and private parties only serve to improve cyber awareness and expose areas for growth.

Cyber Resiliency for Weapons Systems Mr. Daniel Holtzman, Technical Director, Cyber, United States Air Force

Mr. Holtzman chaired a three-person track on cyber resiliency for weapons systems with opening remarks on mission effectiveness. Since the incorporation of cyber in some of our nation’s most critical weapon systems, the role of resiliency ever more critical to ensure completion of missions. Mr. Holtzman believes we need increased number of examples of cyber resiliency techniques and technologies incorporated into these systems


DoD Weapon System Engineering for Operations in Contested Cyberspace Environments Ms. Melinda Reed, Deputy Director for Program Protection, Assistant Secretary of Defense, Research and Engineering (ASD(R&E))

Ms. Reed emphasized that, although once the prevue of military IT professionals, cyber security has become now everyone’s business. The challenge is that every service handles it differently because of how government does business, by policy, under different operational constraints.

It is both unrealistic and unnecessary to expect all parts of military systems to be made secure. All parts of military systems cannot From a technology perspective, it is more critical to understand which components are most critical to the success of the mission. By prioritizing according to mission success, those components can be protected through software assurance, hardware assurance, SCRM, anti-counterfeits, and other means. In addition, the same approach should be taken with information that is central to the success of missions. For example, the correct levels of classification, export control information, with information security processes, will go a long way to making military systems more resilient. Ms. Reed reminded the audience that DoDI 5000.02, Enclosure 14, states the acquisition workforce must take responsibility for cybersecurity from the earliest research and technology development through system concept, design, development, test and evaluation, production, fielding, sustainment, and disposal. Once systems are fielded, they become exposed to a changing cyber threat environment and more potential vulnerabilities. Planning for maintaining the cybersecurity of the system must be considered early and throughout the life cycle. It is the program managers (PMs) responsibility to:

• Plan for and implement effective software configuration updates and software management, includes software patch management during sustainment to mitigate newly discovered vulnerabilities

• Plan, define, and document roles and responsibilities in the appropriate logistics documentation, (e.g., software support plan, operational technical manuals, planned maintenance support)

• Conduct periodic reassessments of cyber vulnerabilities to the system and support systems

• Ensure program and system information are protected and cyber vulnerabilities introduced by depot and other sustainment activities are minimized.

DoD began a series of cyber resiliency workshops, each focused on different topics of concern. A sixth was held after the Invitational in August 2018, the focus of which was the engineering workforce. Strengths and gaps were to be identified with recommendations made for skill sets and curricula needs.


DoD is in the process of developing cybersecurity / resiliency requirements. Cybersecurity needs to be considered regardless of whether it is an explicit stakeholder requirement. Cybersecurity requirements derivation should be derived on the environment in which the systems operate – need to be written into understandable statements that can be tested. Security is important – it hasn’t been part of the design considerations we need to start. While documents such as NIST SP 800-53 may provide valuable information assurance guidance, that is input into the overall requirements process, not a replacement for developing security requirements.

Cyber Resiliency for Weapon Systems Col. Ed Masterson, Acting Director, Cyber Resiliency Office for Weapon Systems (CROWS), USAF

Col. Masterson’s opening remarks addressed the Air Force’s Cyber Resiliency (CR) Campaign Plan for Weapon Systems. The contrast between system’s development and system’s operations was highlighted; the government develops systems one at a time, but once developed. These same systems are intended to operate as a system-of-systems. Col. Masterson further explained the existence of numerous processes in acquisitions, operations, infrastructure, etc. The challenge is that at the larger AF level, there are sometimes gaps between the seams of processes.

The Cyber Resiliency Office for Weapon Systems (CROWs) works under two main objectives, requiring different approaches:

1. “Bake-In” cyber resiliency into new weapon systems 2. Mitigate critical vulnerabilities in fielded weapon systems

To accomplish these two objectives, CROWS operates under seven lines of action (LOA). They are:

LOA 1: Perform Cyber Mission Thread Analysis. The challenge is to determine how deep do these analyses need to be.

LOA 2: “Bake-In” Cyber Resiliency. The challenge is how to integrate security engineering in the systems engineering process.

LOA 3: Recruit, Hire & Train Cyber Workforce. The challenge for LOA 3 is the wide diversity of needs in a cyber workforce.

LOA 4: Improve Weapon System Agility & Adaptability. The AF is moving towards open systems so that systems may be changed out with other available systems. With the right requirements, systems should become less vulnerable and more adaptable.


LOA 5: Develop Common Security Environment. One of the questions addressed by LOA5 is how should the AF manage vulnerability information with consistent classification guides? What should the guides look like?

LOA 6: Assess & Protect Fielded Fleet. Along with ongoing field assessments, the AF will coordinate with its test community.

LOA 7: Provide Cyber Intel Support. The AF developed a methodology to ask very specific and actionable questions. In addition, the AF strives to better understand their threats.

In closing, nine recent accomplishments to enhance weapon system cyber resiliency in contested environments were reviewed, as follows:

• Developed a prototype Cyber Mission Thread Analysis (CMTA) Framework to analyze cyber vulnerabilities, risks, and mitigations in a mission context

• Published the System Security Engineering (SSE) Acquisition Language Guidebook, v. 1.1

• Developed a standard entrance/exit criteria for design review • Defined basic to advanced CR training for acquisition workforce • Developing adaptable standard open architectures • Established a common weapon system security classification guide • Leveraged rapid prototyping • Enhanced processes for analysis of intel/threat data • Established FFRDC/UARC and industry partnerships

Applying Cyber Prep 2.0 and Cyber Resiliency to Build Out a Risk Universe Mr. Jim Mailliard, Vice President, Cyber Security Governance, Risk and Compliance, Elsevier

Mr. Mailliard noted that we know information about only a small part of the universe. We want to fill the gaps with information from tools (e.g., NIST, OWASP, CSA, ISO etc.) We need to think about risks and threats strategically – certain things are for certain environments. One needs to incorporate business into what you are doing – need to blend the terminology between Resiliency and business goals. He noted how he had chosen to use the MITRE Cyber Resiliency Engineering Framework to establish goals. He then worked to intertwin resilience goals with business goals.

• (Anticipate) He rephrased Hygiene as activities for anticipating goal • (Withstand) Many planning activities (e.g., table-top activities) tend to operate as one

event at a time. But in reality, withstand the adversary may employ a chain of events,


or series of a parallel event, or will be looking at the opportunities for the adversary that the one event may open

• (Recover) He recommended not only planning for what is needed to re-establish basic functionality, but what is needed to fully recover.

• (Adapt) He recommends working for continuous improvement, strive to be a step ahead of the adversary.

He explained his vision of the linkage between asset management and risk management. He sees the asset management, which is focused on building and re-configuration, being drawn from policy, controls, and defined standards, and risk management focused on threat assessments (especially external), risk registration, and policy exceptions, utilizing, as appropriate, information and findings from business continuity and disaster recovery efforts. Tools would be employed to assess the ongoing enforcement status of the implementations (e.g., techniques) relative to the risks and where there are opportunity gaps.

He went on to explain how he used MITRE’s Cyber Prep 2.0 framework to help support the risk portion of his methodology. Cyber Prep is a threat-oriented approach that allows an organization to define and articulate its threat assumptions, and to develop organization-appropriate, tailored aspects of a preparedness strategy. It analyzes the threat from the perspective of the adversary’s intent, targeting and capabilities, and the defender’s preparedness with regards to its governance, operations and architecture & engineering perspective. In so doing it identifies gaps and guidance of what the defender can do to address the identified threats. Mr. Mailliard explained how he was using the Cyber Prep framework to help inform the inherent risk portion of his assessment. He explained how understanding the inherent risks helps calculate the overall residual risk which he defined as the inherent risk minus the controls’ reduction (those identified controls/mitigations which help reduce the risk).

Product Vendor Talks The presentations ended with each of the dozen vendors introductory remarks and short discussions on their resiliency-related products.

Section 4: Break Out Sessions, Tuesday, May 8th Breakout Session: NIST SP 800-160 Volume 2 The session was run by Dr. Ron Ross of NIST with Mr. Rich Graubart and Ms. Deb Bodeau of MITRE.

Goal The track’s purpose was to hear feedback on the initial public draft (IPD) of NIST SP 800-160 Volume 2: System Security Engineering – Cyber Resiliency for the Engineering of Trustworthy Secure Systems. NIST SP 800-160 Volume 2, was released on 21 March 2018 for comments and is the first national guideline which


focuses on cyber resiliency. The breakout session was intended as a vehicle for individuals to provide direct comments and questions to the primary authors.

Discussion / Observations The session began with an introduction of session participants which included representatives from JHU/APL, the NY Federal Reserve Bank, the Board of the Federal Reserve, the Department of State, Indiana University, US Navy, and MITRE. As part of the introduction, participants were asked why they were interested in the document and what problem in their organizations and systems they hope to resolve from the document. The answers were diverse and included:

• How to implement cyber resiliency • How cyber resiliency might fit into the concept of NIPRNet/SIPRNet Cyber Security

Architecture Review (NSCSAR) • How to report on an organization’s degree of cyber resiliency • How to assess cyber resiliency • How to improve the operational cyber resilience of a sector, possibly for issuing

guidance

The first part of the discussion was based on questions posed by Dr. Ross and his team as to what information (e.g., concepts, practical guidance, linkages to other NIST publications) the audience needed, what portions worked well, and what needed more work. The resultant responses were somewhat disjointed reflecting the diverse audience.

There were questions as to the purpose of NIST SP 800-160 Volume 2 relative to NIST SP 800-53. The authors noted that NIST SP 800-160 Volume 2 is intended to act as a bridge between the SSE world and the world of the Risk Management Framework (RMF), including NIST SP 800-53 and NIST SP 800-37. They explained how ideally, based on an organization’s mission, stakeholders would identify the relevant cyber resiliency goals, and then objectives that derive from the goals; systems security engineers would then identify the techniques and approaches that derive from the objectives. Those in turn would point to the NIST SP 800-53 controls that could be used to support the selected goals, objectives, techniques and approaches. One of the related questions on controls was whether resiliency was fully covered in NIST SP 800-53, that is, do the controls address all the techniques and approaches noted in NIST SP 800-160 Volume 2. The response by the authors was that (at least in Rev 4 of NIST SP 800-53) all the techniques were reflected in one or more controls, but the same review had not been done for Rev 5 of NIST SP 800-53, nor had a review been done to ensure all the cyber resiliency approaches were captured in NIST SP 800-53 for either Rev 4 or Rev 5.

The discussion on controls led to a conversation about the issues of controls and baselines and tailoring vs. compliance. It was suggested that there is a cultural problem of people employing baselines in a checkbox approach rather than as first step in a broader risk management approach. Dr. Ross used this discussion as an opportunity to discuss the “Risk Management Framework (RMF) 2.0” document that was being released the next day. Dr. Ross explained that


the “RMF 2.0” document would allow greater flexibility in control selection. He also suggested that the individuals in the room might wish to craft some text to be placed in NIST SP 800-160 Volume 2 to address the cultural compliance mindset problem.

In general, the participants concluded that they liked the structure of the document. However, the sheer size of it makes it somewhat hard to process. It was agreed that document comprehension would be greatly enhanced by more use cases and vignettes as well as an index to those use cases.

One question was raised concerning why the document was so heavily focused on addressing the APT. One of the authors explained that the constructs of NIST SP 800-160 Volume 2 could work against any threat or disruption to cyber assets upon which a mission depended. The document focused on the APT because, in the words of one author, it is a very real threat which is essentially “eating our lunch”. In addition, there is a lack of unclassified national level guidance on dealing with the APT. The participants seemed to agree with that reasoning, but felt it needed to be more explicitly articulated in the document.

Some participants also attended the Cyber Resiliency Invitational’s tutorial the day before or were familiar with various MITRE Cyber Resiliency Engineering Framework (CREF) documents. Those individuals commented on their desire to see some of the other MITRE CREF elements included in an updated NIST SP 800-160 Volume 2. Some possible additions explicitly suggested were:

• Assessment methodology (process of how to assess the cyber resiliency needs of an organization);

• Mapping of cyber resiliency constructs (e.g., techniques) against various stages of the cyber-attack lifecycle /kill chain;

• Identifying dependencies between cyber resiliency and cyber security hygiene (i.e., what parts of cyber security hygiene are needed to support cyber resiliency, and what parts an organization might eliminate if cyber resiliency measures are in place); and

• Measurements of cyber resiliency.

Challenges The discussion highlighted several challenges for authors and users of the draft document.

One challenge that became apparent was that the large amount of information in the document makes it hard to read. This may be partially due to most readers’ lack of awareness of the more fundamental aspects of cyber resiliency. Trying to address that problem has given the document something of a tutorial flavor.

Another challenge relates to the document’s intended audience. The document has portions written for the system security engineer. Other portions seem written for architects and the acquisition community. Given the diverse community that will be using this document and the breadth of the information it contains, the reader has difficulty discerning how the document can


be best applied for a given need. An issue related to this challenge is how to make the document relevant for different environments (e.g., medical, enterprise IT, weapon systems).

A challenge resulting from the recommendations of this breakout session is how to incorporate additional material that was requested (e.g., material on assessment methodologies, metrics, linkages to related efforts such as NSCSAR) into the document without making it too overwhelming.

Recommendations/Way Forward 1. The next version of NIST SP 800-160 Volume 2 should incorporate use cases to

maximize understanding and utility. These should cover a diverse set of environments including; power, banking, medical sector, and weapons systems.

2. The use cases should capture threats, mission impact, relevant CREF design principles, goals, objectives, techniques, approaches, mapping back to NIST SP 800-53 controls when possible.

3. To help convey to different classes of users how cyber resiliency is relevant to those individuals and the jobs they need to do there should be discussions regarding linkages of cyber resiliency to different user roles – either as part of the use cases, or as a separate part of future versions of NIST SP 800-160 Volume 2.

4. In order to reduce the document’s complexity and burden on the reader, the document should be examined to identify what elements are key to the thrust of the document and which portions, if any, should be transferred and covered to another document. One possibility might be putting some material in a NIST Interagency Report (NISTIR).

5. The linkages between this document and the documents with which it connects (e.g., NIST SP 800-160 Volume 1, NIST SP 800-53) should be identified, checked and validated. That is, all the cyber resiliency relevant processes identified in NIST SP 800-160 Volume 1 need to be examined and mapped to the cyber resiliency constructs, and all controls in NIST SP 800-53 Rev 5 must be examined to ensure that all the cyber resiliency techniques and approaches are fully covered.

Breakout Session: Resiliency without Detection - Mini-Table-Top Exercise (TTX) This session was developed and led by Mr. Shane Steiger, Principle Cyber Security Engineer, Cyber Operations, The MITRE Corp.

Mr.Steiger led a breakout session on Day 1 of the Invitational which was a “Mini-Table-Top Exercise.” Mr. Steiger explained that he entitled this a mini exercise because it is relatively small, compared to others he has participated in and because it is solely card-based, without a randomizing event component.


The agenda was comprised of three phases followed by a conclusionary readout. Specifics follow.

Phase 1, 45 minutes Mr. Steiger reviewed the background of his career. It served as an introduction to cyber resiliency and the importance of the Cyber Resiliency Engineering Framework (CREF). While reviewing the CREF, resiliency goals, objectives, and techniques were discussed. Since most of the 25 participants were already familiar with the specifics of the CREF, Mr. Steiger used this time to for perspective and highlighted key points. First, Mr. Steiger discussed the defender’s overall goals to achieve objectives via the appropriately balanced use of techniques. He went on to stress that the relationship between techniques is sometimes supportive depending on techniques utilized; in other cases, techniques may conflict with one another. The use of any one set of techniques is a balancing act: the cost to deploy all techniques is prohibitive while the use of too few techniques creates gaps in the ability to achieve the objectives and thereby resiliency goals. Mr. Steiger proceeded to step through a high-level application of the CREF using the goal of withstand, defined as, “to continue essential mission/business functions despite successful execution of an attack by an adversary.” This goal was linked to the related CREF objective of constrain, (“limit damage from an adversary’s attacks”) and the techniques of non-persistence, segmentation, and substantiated integrity, defined as to generate and retain resources as needed or for a limited time, to separate components, logically or physically, based on trustworthiness and criticality, and to determine that critical services, information stores, information streams, and components have not been corrupted, respectively. Next, participants were assigned to one of five teams, comprised of five members each. These groups served as defenders, blue teams. (Blue teams are the primary audience for using resiliency techniques to engineer and architect solutions both strategically and tactically.) The leaders of the table-top exercise, the white team, attempted to make team assignments by mixing resiliency knowledge levels of the participants. Once in their groups, participants discussed the


scenario they would be facing: achieving resiliency without the ability of detection. Both the benefits and difficulties of achieving this were discussed.

Phase 2, 45 minutes Group-based brainstorming followed. Card sets representing the goals, objectives and techniques of the CREF were distributed to each participant and explained. Groups started to build their own notional architecture by selecting three resiliency objectives, e.g., prevent, constrain, and continue, and three complimentary resiliency techniques, e.g., coordinated defense, deception, and privilege restriction, they thought might work in a zero detect architecture. Mr. Steiger guided the blue teams of the desirable elements and best of breed integrated architectures: micro segmentation, zero trust/minimal trust, moving target defense and deception. Next, Mr. Steiger served as the red team. The assumption of the blue teams’ architectures was that they would experience LDAP (Lightweight Directory Access Protocol) injections, commonly used to exploit web applications. Such attacks could reveal sensitive user information or modify information represented in LDAP data stores. A series of three injections were levied on the teams in a serial manner, giving 10-12 minutes between each for teams to react by selecting techniques from their card decks to defend against the attacks. The injections were: DDoS, persistent and adaptive attacker, and then finally a zero-detect adversary. As a lessons-learned activity, Mr. Steiger presented a mapping of resiliency techniques to the three injects after the teams developed defenses. For example:

Taken from CREF. Mapping Resiliency Techniques to Objectives.


Inject Resiliency Techniques 1 - DDoS Dynamic Positioning

Deception Non-Persistence Redundancy Unpredictability

2 – Persistent and adaptive attacker Diversity Segmentation/Isolation Dynamic Positioning Deception Non-Persistence Redundancy Unpredictability Coordinated Protection* Adaptive Response*

3 - Zero-detect adversary Diversity Substantiated Integrity Deception Non-Persistence Redundancy Realignment Adaptive Response*

Phase 3, 45 minutes During Phase 3, each group’s responses were mapped against the injects. Similarities and differences were discussed. Groups reflected on missing techniques that would have helped in defending their architectures and techniques that may have conflicted with each other. Next, Mr. Steiger reviewed the technologies that exist to support the more successful resilient architectures along with control surfaces/services most amenable to zero-detect resiliency: virtual, non-persistent, and wide scalable. Readout, 30 minutes As a conclusion, individual blue teams were brought together to discuss the following:

• What have participants learned from one another? • What did the white team learn? The red team? • What is the value of resiliency without detection? • What were the difficulties in building resiliency in the architecture under these types

of attacks? • Looking beyond the table-top exercise, what will participants try to build in their

organizations?


Break out Session: Operationalizing Cyber Resiliency The session was led by Ms. Emily Frye, MITRE, and included a presentation by Mr. Scott Musman, Principle Cyber Security Engineer, MITRE Goal As the cybersecurity community has experimented with operationalizing resiliency analysis methods and techniques, some common benefits and challenges have emerged as lessons learned. The goal of this track was to capture some of the best developments and crucial observations to share with the community for application to future resiliency efforts.

Discussion / Observations To kick off the session and stimulate discussion, Mr. Musman provided an introductory briefing entitled "Operationalizing Resilience: Requirements, Obstacles, and Anecdotes.” As part of his presentation, Mr. Musman:

1. Placed the operationalizing cyber resilience discussion in the context of mission risk management, in that cyber risk should be considered as one specific contributor to risk to mission. Conversely, cyber resilience should be focused on measurably improving favorable mission outcomes, not just improving the resilience of cyber systems.

2. Described a variety of available methods and tools for assessing mission resilience to cyber threat. He described in detail the Cyber Mission Impact Assessment (CMIA) method that has been applied to multiple operational systems, and the automation of the CMIA method as provided in the "Cyber Security Game" (CSG) prototype.

3. Identified key areas of mission analysis that have a high return on (analysis) investment. For instance, he noted that experience shows operational tasks and functions supporting core mission activities change very slowly over time, as compared to systems architecture. Therefore, a deeper, more thoughtful analysis of core mission activities are highly reusable and worth the investment. Conversely, analysis of architecture/design and interdependencies in technical systems should be done only where immediate and/or high returns are expected, because the technical implementations change more rapidly.

A session participant noted that MITRE's categorization of cyber impact effects (DIMFUI) can be mapped to Parker's Hexad of Loss Scenarios, as shown below.


DIMFUI A. Temin and S. Musman, "A Language for Capturing Cyber Impact Effects, MTR 100344, PR 10-3793," The MITRE Corporation, Bedford, MA, 2010

Parker’s Hexad of Loss D. Parker, Fighting Computer Crime: A New Framework for Protecting Information, John Wiley & Sons, 1998. ISBN: 9780471163787. Chapter 10.

Degradation Utility Interruption Availability Modification Integrity Fabrication Authenticity Unauthorized Use Control (or Possession) Interception Confidentiality

Table 1. DIMFUI vs. Parkerian Hexad

Issues and Challenges

1. Metrics: “How do we rack/stack cyber-related issues against that of other threat areas (e.g., physical, environmental) if we only have relative comparisons against other cyber threats?”

2. Attendee from DHS critical infrastructure and HVAs: “We've done ourselves a disservice by mystifying the cyber effects actors/methods. Need to be specific about our assumptions.”

3. How do we understand and calculate the ROI on cyber investments to protect? From CDR, R. Salvia, Navy, “We have to identify the probability that this fix will reduce the risk by this much, therefore it is worth the investment.”

4. Mr. Musman commented on model-based systems engineering: “We're just starting to figure out how to get cybersecurity and cyber effects into these models. You can approach it as this is just another risk source.”

Following the presentation, session attendees participated in one of two group discussions. The first group focused on identifying helpful (positive) developments in resiliency management and deployment; the second focused on identifying the most critical challenges to operationalizing resiliency. Positive Developments

1. Cyber Table-Top Exercises (TTX) - the group observed that the introduction and use of resiliency frameworks and techniques to the practice of Cyber TTXs was a very effective way to demonstrate the value of resilience investments to stakeholders by stepping through cause, effect, and mitigation scenarios. Guidance from OSD DT&E on conducting TTX is published and updates are underway, including some incorporation of resilience concepts.

2. Both Navy and Air Force have incorporated resilience to (cyber threat) as a topic in their TTX guidebooks as well. The TTX process is effective because the documentation often does not reflect operational truth, but the TTX engages the


operations personnel in the scenario. By doing so, the TTX can reveal operational truth and enable brainstorming on alternative resilience options, additional effects, and ultimately, awareness of mission impacts related to resilience. The AF is actively using the results of the resilience scenarios exercised to further influence and drive the test and evaluation requirements for targeted programs.

3. Commercial and cloud service providers have pushed the envelope in some resilience areas, such as active monitoring and dynamic positioning. Solutions like ChaosMonkey (Netflix) were mentioned as good examples of techniques for building in random, persistent, component failures into systems to ensure resilience is constantly tested.

Challenges 1. Legacy systems are often resistant to many of the available architectural or

technological resilience techniques suggested in the resiliency frameworks. This is often due to aging technical infrastructure, lack of supplier motivation to revisit architectural issues, and lack of skilled technical staff for re-engineering older systems.

2. Impact on Operations - Does it make it too hard to do what you want to do? What are the human factors involved? Are there analog equivalents to cyber?

3. Design artifacts that drive resilience, such as requirements, lifecycle 4. Clearly expressing resilience need to develop, to test, in SOW 5. Other - Fallback failover processes, ROI, metrics

Section 5: Break Out Sessions, Wednesday, May 9th Using Cyber Resiliency to Mitigate Adversary Actions The track was run by Mr. Shane Steiger of the MITRE Corporation, with presentations by Mr. Steiger and Mr. Dan Fitzpatrick, both from MITRE.

Goal Analysis of adversary actions and cyber resiliency are often considered two distinct concepts. This track was intended to challenge this belief by examining how adversary focused frameworks can be used in conjunction with cyber resiliency solutions to help counter advanced cyber-attacks.

Discussion / Observations The discussions were focused around two briefings that employed two different adversary-oriented frameworks and how they are linked to cyber resiliency. The two adversary-oriented frameworks employed were the MITRE’s Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK™)) and the MITRE Adversary Cyber Resilience (ACR) Framework. Mr. Shane Steiger of MITRE led the discussion of employing the former, Mr. Dan Fitzpatrick of MITRE led the latter discussion.


ATT&CK and CREF Discussion As part of his presentation Mr. Steiger:

• Reviewed the foundational elements of Adversarial Tactics Techniques & Common Knowledge (ATT&CK™)

• Discussed an Industrial Control Systems Architecture and where attacks would be focused and an ICS ATT&CK Model

• Reviewed foundational elements of Cyber Resiliency Engineering Framework (CREF).

Mr. Steiger then explained how adversaries are using many of the constructs articulated in the CREF (even if they do not realize that are using them). In effect, the CREF which is a framework focused on maintaining an organization’s mission effectiveness despite adversary’s having compromised portions of the infrastructure, is being flipped by the adversary, who is focused on maintaining their presence in an organization’s despite efforts by the organization to dislodge them. Mr. Steiger also discussed how adversaries use ATT&CK and CREF and how defenders use ATT&CK and CREF and how they differ. For example, he noted how attackers generally emphasize the CREF goals of Anticipate and Adapt, while the Defenders emphasize the goals of Withstand and Recover.

Next, three real world incidents were discussed: Regin Malware, the attack on Ukrainian power systems in 2015, and the CrashOverride Malware event of 2016. For the last two Mr. Steiger looked at them from both the attacker and defender perspective. As part of that discussion, he provided a discussion of which cyber resiliency techniques could counter the actions of the adversary (that were identified via ATT&CK analysis) and specified technologies that could be deployed to have countered the adversary actions.

Furthermore, Mr. Steiger discussed the duality of the attacker and defender’s actions where the defender can pick from the attacker techniques and the attacker is picking from the defender


techniques. He noted that due to the duality of choice, this puts the CREF in a central relationship to the attacker and defender as part of the pictures from the slides. Following up on that point, the speaker also speculated about the use of the CREF to describe/identify potential adversary weaknesses. He hypothesized that the CREF analysis which identifies dependencies and conflicts among CREF techniques (i.e., which CREF techniques depend upon or support other techniques, which technique conflict with/undermine other techniques) could be used to counter adversary actions, as the adversary employs CREF techniques.

ACR Discussion The discussion began with an explanation as to why cyber resiliency is needed. Mr. Dan Fitzpatrick explained that adversary-driven cyber resiliency is a systems engineering methodology for prioritizing cyber resiliency techniques against high end adversaries. In support of that one needs to understand the adversary’s objectives and opportunities at various points in the cyber-attack lifecycle. Mr. Fitzpatrick described the ACR attack matrix and importance of rating an attacks level of difficulty over the attack-lifecycle as well as rating the level of impact that the attack will have. The ACR methodology takes the sum of each of the two values and divides impact by difficulty to determine the adversaries return on investment (ROI). After that an organization can analyze the attack matrix data for trends and patterns; determining what types of attack give the highest ROI and what points of attack provide the adversary the greatest ROI. Presumably attack types attacks and points of attack that provide the adversary the greatest ROI are the ones a defender should be most focused on defending against.

Mr. Fitzpatrick then provided an example of how the CREF can be applied to the attacks highlighted by the ACR. In the example, the ACR methodology identified Flooding as the attack that provided the adversary the greatest ROI. Dan showed how one could identify the CREF goals, objectives, techniques, approaches and technical implementations (of the approaches) to mitigate the attack. He then showed how the goals, objectives, techniques, approaches and implementations that were best suited to counter the flooding attack could themselves be assessed to provide a Defender ROI. That in theory could help inform the defender of which countermeasure is best suited.

Mr. Fitzpatrick then proceeded to discuss five key resiliency principles that have emerged from applying ACR several times. The key principles to disrupting an adversary’s activities were:

• Minimize adversary exposure (e.g., keep systems off (non-persistent services) • Confuse the adversary (e.g., non-persistent services, apply diversity, avoid predictable

solutions) • Maximize segmentation (e.g., use physical or logical partitioning) • Use special sauce (e.g., where feasible) • Architect for extensibility (e.g., adjust to evolving threat)

Mr. Fitzpatrick continued to explain how this approach has been applied in several programs of varying scale and in different phases of their lifecycles to inform different types of decisions.


• Develop a prioritized list of countermeasures designed to mitigate the most likely attacks by advanced adversaries;

• Inform red team engagements and helped aid in the development of next generation systems; and

• Influence future defensive investments.

Challenges ATT&CK and CREF Challenges

The greatest challenge might be the need to how to break out of the stovepipes of the ATT&CK and CREF communities. There is a need for each community to determine how to look beyond their own perspective and see how the frameworks can work together.

Another challenge is to validate some of the assumptions articulated in the presentation (e.g., can one undermine adversary’s actions via analysis of CREF dependencies).

ACR Challenges ACR has been employed in a limited number of situations. Where it has been employed are sensitive applications, therefore limiting the information about ACR application that can be shared. Given the limited number of applications of ACR impacts the ability to validate or refine the methodology employed. Possible challenges (or refinements) to consider include whether the ROI formulas are correct as is, or is some modification (e.g., weighting) needed.

Recommendations/Way Forward For both approaches the way forward involves greater application of the briefed methodologies.

Greater analysis of actual incidents via ATT&CK and CREF is needed to provide more concrete evidence to support Mr. Steiger’s hypothesis that the two frameworks can be used to better understand how to defend against the adversary. Any identified incidents would need to be assessed via both ATT&CK and CREF.

Work should also be applied to attempting to validate Mr. Steiger’s hypothesis that dependencies and conflicts among CREF techniques could be used to counter adversary actions, as the adversary employs CREF techniques.

With regards to ACR, applying it to more environments will help validate (or disprove) some of its findings. In addition, greater application of ACR will potentially provide insights that can allow for massaging ROI calculations.


Cyber Resiliency in Weapons Systems Track Chair: Mr. Daniel Holtzman, USAF Cyber Technical Director

Goal This session focused on the challenges of metrics and measures for resiliency and communicating resiliency concerns and requirements.

Observations The session began with observations on the current environment and prior efforts at both security and resiliency. First, the speaker asserted that no matter how much security is done, ongoing challenges will always be changing, complex regulations, adequate budget considerations, skilled staffing considerations, and above all, the threat changes in complex environments. Second, security professionals have unknowingly reinforced a compliance mindset by penalizing organizations when they are out of compliance. Third, security attestation is very difficult when production of components is done in countries, e.g., China, which may be adversaries. Finally, in discussions of security and cyber resiliency, there is often no clear differentiation between component and system levels. This lack of distinction makes it difficult to communicate requirements and dependencies clearly.

Starting from these observations, session participants offered their thoughts on what must be done to implement resiliency. Observations fell into four categories: people, existing processes, communication, and frames of reference. Observations are summarized below.

1. People: It was asserted that people are key to the solution. Participants further noted that system users are doing things with systems for which they were not originally designed, thus necessitating proper, up to date training. In addition, policies and procedures need to be in place.

2. Processes: Efforts to incorporate cyber resiliency should be leveraging both existing process such as safety processes and lessons learned from those processes. Safety processes focus on hazards and safety constraints rather than threats which are much more transient and tactical.

3. Communication: Communication is key to successfully incorporating cyber resiliency into weapons systems. Two success stories were shared, both concerning the nature of communicating risk. One identified the need to share information up to the mission or platform level. It is successful when cyber experts can provide a judgment with confidence. This is a difficult to scale. The success regarded a piece of vulnerable software. The risk was not to the system owner but to the overall mission owner. Providing the information to the mission owner enabled that person to make a better risk decision and forced the fix that needed to happen.

4. Frames of Reference: It was noted that framing the problem properly is critical. This is true both in communications as well as in implementation and measurement of cyber resiliency. The breadth of the different aspects that must be addressed ranges from control systems such as power and fire suppression systems to supply chain risk


management. One key to addressing this breadth is to recognize that weapons systems are cyber physical systems with layers: physical, control, cyber, users, and system. Additionally, there are three different lenses when looking at recommendations: one for legacy systems, one for new systems, and one for sustainment efforts. With this complexity, it is critical to cyber resiliency to identify the mission and the boundaries of the system.

As part of the discussion, Mr. Holtzman proposed five possible topics that the group could discuss either in parallel or sequentially:

• Metrics and measures • Communications • Risk identification • Legacy, new sustainment • Science & Technology Gaps/Needs

The consensus was to tackle the topics as a group. The group started with Metrics and Measures with the intent to cover the remaining four topics as well. However, the nature and challenge of the first topic consumed the entire meeting.

Metrics as a tool for communicating status and progress must be relevant to the environment. For example, system assurance at an engineering level means functions are going to be available as needed. At a mission level, mission assurance means bombs on target. These are very different meanings of assurance.

There has been about four years of work in the metrics area. Metrics work has been attempting to measure how well security has been incorporated into programs. In the acquisition phase the metrics have tried to determine if the right processes were built in. In the operations phase the metrics asked how well the technical orders worked and how well they were followed.

One approach is to start at the system level and build up to the mission level. It was noted that there are two types of measurement – direct measure of mission effectiveness and indirect measures which are those correlated with the cyber security or cyber resiliency of the system. One way to link these is to use outcome as a measure of behavior. There are good behaviors and bad behaviors (i.e. adverse effects). The good behaviors are captured in the functional requirements and the path through cyber that causes these behaviors. Problematic behaviors can be identified by modeling all behaviors and testing them using threats in context of a given mission. It was noted that there is no single set of metrics. Metrics are highly contextual; the challenge is finding the appropriate context. It was noted that in many instances the bad things one needs to measure at the mission level from a cyber resiliency perspective are the same things one needs to measure from a safety perspective. The difference is that often that the mitigations one needs to counter the bad things that due to natural disasters are different than mitigations to counter bad events due to cyber directed events.


As a result, the measurement of the mitigations will be different as well. This is important because while measurements cannot prevent bad things from happening, they will help identify the right/effective mitigations.

Challenges Legacy systems are particularly challenging because the ability to understand them is hampered. Documentation may not match actual configurations of fielded systems. Systems may be modified for performance reasons and the way they will respond in an adverse cyber situation is not documented. Maintainers are usually doing their best to keep older platforms operational. This sometimes means they implement changes without understanding of collateral damage. The two challenge questions in this area are: How do we get an understanding of legacy systems that is realistic? How do we gain and communicate an understanding of how maintenance changes impact missions as a whole?

A second challenge area is defining what needs to be measured. Two challenge questions in this area are: How can cyber situational awareness be measured on systems? How are measurements rolled up to the mission level?

Agile System Development was identified as a challenge to cyber resiliency. While they are not mutually exclusive, the way that it is being implemented impairs the ability to create resilient systems. Anything agile needs more robust system engineering than we have. The challenge question for this concern is: What guidance can be provided to incorporate cyber resiliency into agile system development?

Way Forward While the discussions in this breakout session did not lead to specific guidance in addressing the challenges, there were many who came away with a new understanding of the challenges themselves. There was also general agreement that this discussion needs to continue, and Mr. Holtzman will be connecting with the group in the coming months to do so.

8th annual secure and resilient cyber architectures

Documents