1 HPAC ENGINEERING NOVEMBER 2016

Your Building Control Systems Have Been Hacked.

Tactics, techniques, and procedures for detecting, responding to, and recovering from a cyber attack

Now What?

By MICHAEL CHIPLEY, PhD, GICSP, PMP, LEED AP; DARYL HAEGLEY, OCP, CCO; and ERIC J. NICKEL, RCDD, GICSP, CPT, CEH

NETWORKED CONTROLS

gies (OT) and use a combination of traditional informa-tion-technology (IT) protocols—Transmission Control Protocol (TCP) or User Datagram Protocol (UDP)—and control systems with unique protocols (e.g., Modbus, BACnet, LonTalk, DNP3 [Distributed Network Protocol]) to communicate with sensors, devices, and actuators.

BCS are an easy target for hackers and people with malicious intent because they are unsecured by design, the desire being remote access and an ability to operate with minimal staffing. With smart buildings, instead of several vertical chases connecting telecommunica-tions rooms and redundant communications and power pathways, the trend is to collapse IT and OT traffic onto a single fiber; use Power over Ethernet (PoE) switches to provide low power voltage to OT devices, such as card readers, magnetic door locks, closed-circuit televi-sion, and medical devices and equipment; and eliminate redundant pathways to gain additional rentable floor space. In most organizations, the IT department now is responsible for converged infrastructure, yet has no to little training on detecting, removing, and recovering from malware directed at BCS.

Building owners and managers, facility engi-neers, and physical-security specialists, be warned: Building control systems (BCS) are now squarely in the sights of nation-state

and criminal hackers. With the increased likelihood of BCS being attacked/exploited, U.S. Cyber Command (USCYBERCOM) developed the document “Advanced Cyber Industrial Control System Tactics, Techniques, and Procedures (ACI TTP) for Department of Defense (DoD) Industrial Control Systems (ICS)” (http://bit.ly/ACI_TTP). (Note: The use of the word “industrial” can be misleading; the ACI TTP can be applied to any control system.) This article discusses use of the ACI TTP for detecting, responding to, and recovering from a cyber attack.

BCS BasicsBCS, such as building-automation systems (BAS),

energy-management systems, physical-security access-control systems, and fire-alarm systems, are hacking points into an organization (Figure 1). Such control systems often are referred to as operational technolo-

Michael Chipley, PhD, GICSP, PMP, LEED AP, is a consultant to federal agencies and private-sector clients; contributor to National Institute of Standards and Technology (NIST) Special Publication (SP) 800-82 Revision 2 (R2), Guide to Industrial Control Systems (ICS) Security, and the Department of Homeland Security (DHS) Cyber Security Evaluation Tool (CSET); and creator of the National Institute of Building Sciences’ Cybersecurity for Building Control Systems Workshop Series. He can be contacted at mchipley@pmcgroup.biz. Daryl Haegley, OCP, CCO, has been leading efforts to cybersecure U.S. Department of Defense (DoD) facility-related control systems; is a contributor to NIST SP 800-82 R2, DHS CSET, and Unified Facilities Criteria 4-010-06, Cybersecurity of Facility-Related Control Systems; and is a guest lecturer on cybersecuring DoD control systems for National Defense University and The Institute of World Politics. He can be reached at dhaegley@gmail.com. Eric J. Nickel, RCDD, GICSP, CPT, CEH, is a technical expert with over 19 years of experience in design-build project delivery of turnkey secure information-transport systems. He is the information-technology and infastructure director for Chinook Systems Inc., a multidisciplinary facilities-engineering firm focused on life-cycle energy-security solutions. He can be reached at enickel@chinooksystems.com.

BEEBRIGHT/ISTOCK

NOVEMBER 2016 HPAC ENGINEERING 2

operators and providing basic train-ing in typical IT and OT operations; cyber-attack-event detection, miti-gation, and recovery; and forensics. The teams then practiced stopping a nation-state-type attack, recording the procedures they used to defend their control systems.

The teams’ responses became the ACI TTP, which were published in January 2016. The ACI TTP apply to IT systems—business and home—and OT systems of any kind.

The ACI TTP document is divided into essentially four sections:

1) ACI TTP concepts. Chapters 2 through 4 explain the scope, prereq-

uisites, applicability, and limitations of the components of the ACI TTP.

2) Threat-response procedures (Figure 2). These encompass:

• Detection procedures designed to uncover malicious cyber activity as soon as possible. The basic actions involved with detection are routine monitoring and inspection and transition to mitigation pro-cedures. Detection procedures are designed to enable control-system (CS) and IT personnel to identify malicious network activity using official notifications of system anomalies not attributed to hard-ware or software malfunctions.

NETWORKED CONTROLS

BCS can be exploited to gain physical access to facilities and virtual access to traditional IT systems and data and to damage or destroy building equipment, exposing organizations to sig-nificant financial obligations for containment, eradication, and/or recovery in the process.

Finding BCS on the InternetThere are many tools for moni-

toring network security. One of the most powerful is Shodan (www .shodan.io). In the majority of cases, clicking on an Internet Protocol (IP) address will open the login to an operator console. In many cases, the browser is using unencrypted Hypertext Transfer Protocol (HTTP) port 80, rather than encrypted HTTPS (also called HTTP over Transport Layer Security [TLS], HTTP over Secure Sockets Layer [SSL], and HTTP Secure) port 443, meaning the login credentials are being sent as open text across the Internet. BCS should not be exposed to direct Internet connections; they should be in a demilitarized zone (DMZ) with perimeter-protection firewalls, require a virtual-private-network connection, and be sepa-rate from business IT systems.

IT is about data; OT is about controlling machines (Table 1). Increasingly, OT is becoming more IP-based. The Internet of Every-thing, smart grids, smart cities, smart buildings, and smart cars are redefining the boundary between IT and OT. As IT and OT systems converge, so are the vulnerabilities of OT systems as points of entry. Once a hacker enters a system, it is just a matter of pivoting up the network and taking control of OT- and IT-system assets.

The Development of the ACI TTPThe ACI TTP were developed

over 18 months by pairing IT and OT

FIGURE 1. Typical building control systems.

TABLE 1. Information technology vs. operational technology.

3 HPAC ENGINEERING NOVEMBER 2016

Circle xxx

• Mitigation procedures, which involve analysis, response, and a course of action. Normally, all data should be acquired and preserved for further analysis. Sometimes, however, the need to keep a system operational precludes the collection of data. In such cases, the command authority must approve the noncom-pletion of data acquisition. What-ever response a situation calls for, organizations must be prepared, as decisions made in haste could lead to unintended consequences. Therefore, procedures, tools, inter-faces, and communications chan-nels and mechanisms should be tested and in place. Plans should list specific steps to take and iden-tify the necessary personnel by job description. Escalation procedures and criteria must be established and acceptable risks for incident

NETWORKED CONTROLS

FIGURE 2. ACI TTP threat-response procedures.

4 HPAC ENGINEERING NOVEMBER 2016

containment defined. This can be done during annual risk-manage-ment activities.

• Recovery procedures. A cyber-protection team (CPT) from out-side an organization may be called

upon to direct a recovery process. The main focus of a CPT should be to preserve forensic evidence for analysis and to provide technical assistance as needed. If directed, an operator may proceed with recovery without the assistance of an outside CPT. In any event, when-ever feasible, every effort should be made to preserve evidence of a cyber incident for forensic anal-ysis. Forensic-evidence collection for BCS is very difficult and time-consuming, as most building con-trollers do not have logs, are not authenticated, and are on unen-crypted networks.

3) Routine monitoring and base-lining. Capture the “fully mission-capable” (FMC) condition of net-work entry points (e.g., firewalls, routers, remote-access terminals, wireless access points), network topology, network data flow, and machine/device configurations. The FMC baseline is used to determine normal operational conditions vs. anomalous conditions. A recovery jump kit contains the tools BCS and IT teams need to restore a system to its FMC state during mitigation and recovery. Knowing what the recovery point should be is the key to ensuring all known remnants of an attack have been removed from all components of an ICS. This means all hardware and software are configured in accordance with operational requirements and checksums and hashes are in con-formance with vendor specifica-tions.

4) Reference materials. To fur-ther enhance the ACI TTP as a tool, operators are encouraged to refer to additional resources provided by the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) and the National Insti-tute of Standards and Technology (NIST) Special Publication (SP) 800 Computer Security series.

NETWORKED CONTROLS

FIGURE 5. Malware can be quarantined using antivirus tools such as Malwarebytes.

FIGURE 4. ACI TTP for suspicious software/configurations: Use jump kit to remove virus/malware from server/workstation, perform integrity check.

FIGURE 3. ACI TTP Enclosure A: Detection Procedures, server/workstation anomalies.

NOVEMBER 2016 HPAC ENGINEERING 5

Example of Server/Workstation Anomalies

To illustrate the use of the ACI TTP, we are going to use an exploit scenario created by ICS-CERT us-ing Havex malware (http://bit.ly/ICS_Alert). ICS-CERT described the vulnerability and how to mitigate it and provided a YARA (http://bit.ly/YARA_tool) signature, used to iden-tify and classify malware. F-Secure provided additional research details and code analysis of the malware, showing file names and locations where the malware was installed on exploited machines (http://bit.ly/F-Secure_weblog).

Our analysis of Havex sample codes uncovered the malware’s ICS/SCADA (supervisory control and data acquisition) “sniffing” behavior. The command-and-control (C&C) server instructed infected comput-ers to download and execute further components, one of which appeared very interesting. While analyzing that component, we noticed it enu-merated the local area network and looked for connected resources and servers. We then noticed it used Microsoft (MS) Component Object Model (COM) interfaces (CoInitial-izeEx, CoCreateInstanceEx) to con-nect to specific services. To identify the services in which the sample was interested, we simply searched for the identifiers seen above, which told us what kind of interfaces were being used. A bit of googling gave us these names:

• 9DD0B56C-AD9E-43EE-8305-487F3188BF7A = IID_IOPCServerL-ist2

• 13486D51-4821-11D2-A494-3CB306C10000 = CLSID_OPC-ServerList

Note “OPCServer” in the names. There were more clues—the strings found in the executable made sev-eral references to “OPC”—pointing in the same direction. OPC stands for OLE for Process Control. It is a

standard way for Windows appli-cations to interact with process-control hardware. Using OPC, the

malware component was gathering details about connected devices and sending them back to the C&C for

NETWORKED CONTROLS

FIGURE 6. ACI TTP server/workstation-integrity checks.

FIGURE 7. ACI TTP server/workstation process check.

FIGURE 8. Sysinternals server/workstation process check.

6 HPAC ENGINEERING NOVEMBER 2016

attackers to analyze.MS operating systems are used

in over 90 percent of BCS deploy-ments. An analysis by the DoD found Windows XP on 70 percent or more of BCS (both DoD and commercial). Only systems 5 years old or newer might be on Windows 7, 8, or 10. Very few BCS have been virtualized. They all share the OPC vulnerability.

Figure 3, taken from the ACI TTP document, shows server/work-station anomalies. In the case of A.2.4—suspicious software/configu-rations—the user is directed to the detection procedure on Page A-8 (Figure 4). The procedure instructs the CPT to perform a virus/malware quarantine and removal function, capture forensics information, and document the actions in a security log.

Malware can be quarantined and removed using antivirus tools such as Malwarebytes (Figure 5). Hackers like to install malware into memory, which makes the malware harder to find.

Next, users are advised to run server/workstation-integrity checks (Figure 6).

To check for processes that do not appear legitimate, launch MS Sysinternals (figures 7 and 8). Check Sysinternals Autoruns, Pro-cess Explorer, and Process Moni-tor; look for high central-process-ing-unit (CPU) and memory usage; and compare against process and threads (Figure 9).

If a server/workstation has been exploited and antivirus/malware-removal efforts have been unsuc-cessful, the ACI TTP advises users to proceed to recovery procedures (Figure 10).

M a n y B C S d o n o t m o n i t o r underlying infrastructure. For checking firewalls, networks, and applications for suspicious threads, processes, and CPU and memory

usage, a number of tools, such as GlassWire (Figure 11) and the Sysinternals suite combined with the MS Enhanced Mitigation

Experience Toolkit (EMET), are available.

Forensics data-gathering tools, such as OSForensics (Figure 12) and

NETWORKED CONTROLS

FIGURE 10. ACI TTP server/workstation recovery procedures.

FIGURE 9. ACI TTP server/workstation log review.

NOVEMBER 2016 HPAC ENGINEERING 7

Mandiant Redline, create a forensics copy of drives and perform detailed computer-usage analysis.

ConclusionThis is just a snippet of the

effort required to actively defend a BCS. Unfortunately, at this time, there are very few fully automated tools that can look at a BCS from the end-point device to the server/workstation human-machine inter-face and determine the health and cyber status of the system. As new products that enable machine-to-machine and end-to-end analysis are developed, the level of effort will decrease while overall cyber-security increases. Until that time, BCS owners/operators should familiarize themselves with, budget for, and plan to implement the ACI TTP.

SummaryAt some point in the not-too-

distant future, a major building’s control system will be hacked, leading to significant physical and economic damage. In general, very few organizations have the skills and training to identify, much less respond to, an active attack/exploit situation. The ACI TTP are an organization’s first line of defense, aiding the detection and mitiga-tion of threats and the restoration of a BCS to normal operation. The Whole Building Design Guide cybersecur i ty resource page (http://bit.ly/WBDG_Cybersecurity) provides BCS Cyber 101, while the National Institute of Building Sciences’ Cybersecurity for Build-ing Control Systems Workshop Series (see sidebar “Cybersecurity Workshop Set”) is designed to help architects, engineers, contractors, owners, facility managers, mainte-nance engineers, physical-security specialists, information-assurance professionals—essentially anyone

involved in cybersecurity over a facility’s life cycle—to learn best-practice techniques for better protecting their facilities.

Did you find this article useful? Send comments and suggestions to Executive Editor Scott Arnold at scott.arnold@penton.com.

NETWORKED CONTROLS

FIGURE 11. GlassWire firewall, usage, network, and alerts logging.

FIGURE 12. Data gathering with OSForensics.

On Dec. 6 in Gaithersburg, Md., the authors will present “Your Building Control Systems Have Been Hacked, Now What?,” part of the National Institute of Building Sciences’ Cybersecurity for Building Control Systems Workshop Series.

Intended for building owners, facility managers, engineers, and physical-security, information-assurance, and other professionals involved in the design, deployment, and operation of building control systems, the workshop will provide a combination of classroom learning modules and hands-on laboratory exercises to teach how to detect, contain, eradicate, and recover from a cyber event.

The cost is $300. For more information and to register, go to http://bit.ly/Cybersecurity_workshops.

Cybersecurity Workshop Set

MAR

EAN

DM

ARE/

ISTO

CK