Cybersecurity forEnergy Delivery SystemsMichael Assante & Tim Conway (Under contract to DOE through Idaho National Laboratory)March 28th, 2016

2

Agenda

1. Event deconstruction

2. Mitigations

3. Discussion

UNCLASSIFIED

Ukraine EventDecember 23, 2015

UNCLASSIFIED

4

An interagency team composed of representatives from the NCCIC/ICS-CERT, U.S. Computer Emergency Readiness Team(US-CERT), Department of Energy, Federal Bureau of Investigation, and the North American Electric Reliability Corporation traveled to Ukraine to collaborate and gain more insight.

Mike Assante and Tim Conway as DOE INL subcontractors added to the team by DOE to bring their electricity sector and SANS experience to bear on this critical incident.

This briefing is our post trip report. The mitigation guidance for consideration is our own and is offered “as is”, as general concepts to simply inform thinking

UNCLASSIFIED

Presentation Perspective

5

Geographic Orientation

UNCLASSIFIED

6

Power System Orientation

UNCLASSIFIED

7

Ukraine’s Generation Sites

UNCLASSIFIED

Power System Regions

8

Power System Element: Distribution

UNCLASSIFIED

Source: Modification of an image from the energy sector - specific plan 2010

9

Event Summary

Through interviews, the team concluded that a remote cyber attack caused power outages at three Ukrainian distribution entities (Oblenergos) impacting approximately 225,000 customers

While power has been restored, all the impacted Oblenergos continue to operate in a degraded state

The attack included elements to disrupt power flow and exaggerate the outage by damaging the SCADA DMS and communication infrastructure used to support power dispatching

UNCLASSIFIED

   

   

   

 

    

 

UNCLASSIFIED

12

Attack Steps Summary

• Infect, Foothold, C2• Harvest Credentials• Achieve Persistence & IT Control• Discover SCADA, Devices, Data• Develop Attack Concept of Operation (CONOP)• Position• Execute Attack

- SCADA/DMS Dispatcher Client/WS Hijacking- Malicious firmware uploads- KillDisk Wiping of WS & Servers- UPS Disconnects & TDoS

UNCLASSIFIED

13

Technical Components

• Spear phishing to gain access to the business networks • Identification of BlackEnergy 3 at each Oblenergos• Adversary theft of credentials from the business networks• Use of VPNs to enter the ICS network• Use of existing remote access tools within the environment or

issuing commands directly from a remote station capable of issuing commands similar to an operator HMI

• Serial to Ethernet communications devices impacted at a firmware level

• Use of a modified KillDisk to erase • Utilizing UPS systems to impact connected load with a

scheduled service outage• Telephone Denial of Service attack on the call center

UNCLASSIFIED

14

Phantom MouseRemote Amin Tools at OS-level

Rogue ClientRemote SCADA Client Software

SCADA Hijacking Techniques

SCADA Server

The attackers developed two SCADA Hijack approaches (one custom and one agnostic) and successfully used them across different types of SCADA/DMS implementations at three companies

15

The Ukraine cyber attacks are the first publicly acknowledged intentional cyber attacks to result in power outages. As future attacks occur it is important to scope the impacts of the incident being examined.

Power outages should be measured in scale (number of customers and electricity infrastructure involved) and in duration to full restoration. These incidents impacted up to 225,000 customers in three different distribution level service territories lasting several hours. These incidents would be rated on a macro scale as low in terms of power system impacts as the outage impacted a very small number of overall power consumers in Ukraine and the duration was limited.

We are confident that the companies impacted would have rated these incidents as high or critical to their business and reliability of their systems.

Keeping Perspective

16

Attacks were planned, coordinated, and required high-degree of orchestration

Aggressive development-to-operations cycle Attacks required multiple operators Simultaneous actions & mistakes Multi-staged Kill Chain Multiple attack elements Custom attacks developed Multi-staged attack Attackers achieved objective Targets used different SCAD

What we should understand

UNCLASSIFIED

17

ICS Kill Chain Mapping (Stage 1)

UNCLASSIFIED

18

Stage 1 TTPs

Spear phishing with MS Office Attachments BlackEnergy malware used for initial infection◦ Overlapping C2 servers

KillDisk downloaded and executed manually◦ KillDisk execution on selected Workstations and

Servers Use of company employed remote access tools ◦ Use of legitimate credentials for network access at time

of attack (RDP, RADMIN, VPN) Installation of backdoors

UNCLASSIFIED

19

ICS Kill Chain Mapping (Stage 2)

UNCLASSIFIED

20

Stage 2 TTPs

Lockout of legitimate dispatchers Manual & command operation to trip

breakers Firmware corruption of Serial-to-Ethernet

converters & Substation Devices UPS system outage KillDisk on RTU Local HMI Module◦Windows OS

UNCLASSIFIED

21UNCLASSIFIED

22

Distribution Control Center(s)• Central Office• Branch Offices

110 kV Substation

110 kV Substation

35 kV Substation

HMI workstations (OS-level)

Client-to-Server

AccessibleFirmware Devices

Workstations &Servers

Attack Elements by Location

UNCLASSIFIED

23

Input Output

•No Datao Source problemso Disrupt Com Patho Disrupt AP/Interface

• Invalid Data• Too Much Data

Communications Path

Physical Protection

• Hardware• Firmware• Application Software• Configuration

Electronic Protection

Communications PathAPAP AP AP

•No Datao Destination problemso Disrupt Com Patho Disrupt AP/Interface

• Invalid Data• Too Much Data

Device

Maintenance not operational data input

Malicious Firmware Uploads (Cont.)

Model created by Mark Engles

UNCLASSIFIED

24

Manipulate-to-Disrupt (anti-restore)

UNCLASSIFIED

25 FUNCLASSIFIED

1

2

3

26

How Sophisticated Was It?

FUNCLASSIFIED

27

Rating this Attack

0

0.5

1

1.5

2

2.5

3SOPHISTICATION

CUSTOMIZATION

EFFECT

CONOP

Sophistication

ICS Customization

Effect

CONOP

1

2

2

3

• Some sophistication in the SCADA/DMS hijacking method but the majority of it was not

• Rogue client hijacking demonstrated some customization

• Electricity outage in three service territories restored in hours

• A complex and successful attack plan

Summary

UNCLASSIFIED

28

UkraineIncidents

Human Operators

ICS Infrastructure

ICS Applications

Process & Safety

INCIDENT MAPPING

ElementsHMI InputsAlarmsData

EffectLoss of View (LoV)False Alarms/Suppress AlarmsSpoofed Status, Levels, and ConditionsDenial of Control (DoC)

ElementsServersNetworkWorkstationsOS

EffectModify FilesCorrupt/Destroy DataExhaust Resources/DoSHang ApplicationsHijack

ElementsHMI (Client)SCADA ServersENG WSHistorians/DBsGateways/FEPs

EffectChange Settings & Schedule TasksSpoof Data, Issue Commands (MoC)Delete DataDoS, (DoC)

ElementsControllersComms/IOInstrumentsActuators

EffectChange Settings, Write to Memory Data Destruction Spoof Data, (MoC or MoV)Change Logic, (MoC)DoS/Corrupt Software, (DoC)

SCADA/DMS & Process Elements

UNCLASSIFIED

Guidance & Mitigation ConceptsPublished Advisories and SCADA/DMS mitigations

UNCLASSIFIED

30

https://ics-cert.us-cert.gov/alerts/IR-ALERT-H-16-056-01ADD

FUNCLASSIFIED

ICS-CERT Alert

311

Level 2 NERC Alert (R 2016 02 09 01) that was released February 9, 2016

https://www.esisac.com/api/documents/4199/publicdownload

E-ISAC Alert

32

Control & Operate

VPN Access

Workstation Remote

Ukraine EventSignificant Events based on publicly available reporting.

Ukraine EventSignificant Events based on publicly available reporting.

Credential Theft

Spearphish

Tools & Tech

FUNCLASSIFIED

Attack Elements

IT Preparation• Target selection• Unobservable

target mapping• Malware

development and testing

Sequence Pre Work

• Upload additional attack modules -KillDisk

• Schedule KillDisk wipe

• Schedule UPS load outage

Attack Position• Establish Remote

connections to operator HMI’s at target locations

• Prepare TDoS dialers

Target Response• Connection sever• Manual mode / control

inhibit• Cyber asset restoration• Electric system

restoration• Constrained operations• Forensics• Information sharing• System hardening and

prep

Hunting and Gathering

• Lateral Movement and Discovery

• Credential Theft and VPN access

• Control system network and host mapping

Spear phishing• Delivery of phishing

email• Malware launch

from infected office documents

• Establish foothold

ICS Preparation• Unobservable

malicious firmware development

• Unobservable DMS environment research and familiarization

• Unobservable attack testing and tuning

Attack Launch• Issue breaker open

commands• Modify field device

firmware• Perform TDoS• Scheduled UPS

and KillDisk

Opportunities to Disrupt

12 mo

9 mo

6 mo

hrs.

min

Event

Hrs.

34

• Awareness training• Phishing testing

Training

Spearphish

• Detection Based• Reputation Based

Filtering

• Contested territory• Isolate and control

Anticipated

FUNCLASSIFIED

Spearphish

35

• YARA & AV• Change PW

Remediate

Credential Theft

• Directory Segmentation• Zones of Trust

Defense in Depth

• Normalize net and directory activity

• Alert on the abnormal

Anticipated

FUNCLASSIFIED

Credential Theft

36

• Two factor• Dedicated Tokens

Strengthen

VPN Access

• Jump Host• No Split Tunneling

Trust

• Why is it there• Activate at time of use

Anticipated

FUNCLASSIFIED

VPN Access

37

• Disable remote access• Block at perimeter fw

Harden

Workstation RemoteAccess

• Configure Host FW• Monitor config changes

Manage

• Conservative operations

• Sectionalizing

Anticipated

FUNCLASSIFIED

Remote Access

38

• Logic for confirmation• AOR

App Security

Control and Operate

• Path encryption• Protocol encryption

Communication

• Manual operations• Load Shed

Anticipated

FUNCLASSIFIED

Control

39

• Filter calls by source• Disconnect BCS from net• Disable remote mgmt

Eliminate

Tools and Tech

• Disable remote FW updates• ATS, Backup Gen• Secondary Comms

Device

• Blackstart plans• Islanding• Mutual Aid

Anticipated

FUNCLASSIFIED

Tools and Tech

40

Lessons Learned

TrainingPlanning and AnalysisLoad ShedEOPBlackstart

41 F

• Cyber contingency analysis (continuous analysis and preparing the system for the next event)

• Cyber failure planning (modeling and testing cyber system response to network and asset outages)

• Cyber conservative operations (Intentionally eliminating planned and unplanned changes, as well as stopping any potentially impactful processes)

• Cyber load shed (Eliminating all unnecessary network segments, communications, and cyber assets that are not operationally necessary)

• Cyber RCA (Root Cause Analysis forensics to determine how an impactful event occurred and ensure it is contained)

• Cyber blackstart (cyber asset base configurations and bare metal build capability to restore the cyber system to a critical service state)

• Cyber mutual aid (ability to utilize ISACs, peer utilities, law enforcement and intelligence agencies, as well as contractors and vendors to respond to large scale events)

UNCLASSIFIED

Lessons Learned Translated

42

Component Mitigation N Mitigation N+1 Mitigation N + X

Spear phish Training Filter System Spec

Credential Theft Remediate PW Defense in Depth ProtectionDevices

VPN Access Strengthen Trust RCA / EOP

Workstation Remote Access

Harden Manage Conservative Operations / Sectionalizing

Control and Operate

App Security Communication Manual Operations / Load Shed

Tools and Tech Eliminate Device Black Start / Mutual aid

FUNCLASSIFIED

Prepare to Defend the Effect

UNCLASSIFIED

питанняQuestions

Attack

44

References & Products

NCCIC/ICS-CERT INCIDENT ALERT: IR-Alert-H-16-043-01P UKRAINIAN POWER OUTAGE EVENT, February 12, 2016 (TLP=GREEN)◦ High-level summary of the incident elements◦ Mitigation guidance◦ Detection pointers & indicators (IOCs)

NERC E-ISAC: Mitigating Adversarial Manipulation of Industrial Control Systems as Evidenced By Recent International Events, February 9, 2016 (TLP=RED) ◦ Tactics used by actors with mitigation options

ICS-CERT BlackEnergy YARA signature: https://ics-cert.us-cert.gov/alerts/ICS-ALERT-14-281-01E

Initial Findings of the US Delegation examining the events of December 23rd 2015, Power Point Presentation, February 2016

E-ISAC & SANS Defense Use Case: https://www.esisac.com/api/documents/4199/publicdownload

UNCLASSIFIED

45

Guidance Documents

CEDS

Research & Development

DeploymentIncident Coordination

UNCLASSIFIED

46

Cyber Incident Coordination

Coordinate response with federal and industry partners. Share information and facilitate access to technical sector specific

expertise while ensuring: Unity of effort; and Unity of message

Collaboration with industry for participation in national and regional preparedness projects including cyber exercises. ESCC Playbook Exercise New York State Cybersecurity Exercise (NYSCE) Dams Sector Information Sharing Drill North American Electric Reliability Corporation (NERC) Grid Security

Exercise (GridEx)

CEDS

Research & Development

DeploymentIncident Coordination

UNCLASSIFIED

4747

Office of Electricity Delivery & Energy ReliabilityU.S. Department of Energy

www.energy.gov/oe/services/cybersecurity