How to Fuzz 100+ Industrial DTM Components and Stay Alive

Alexander Bolshev

(@dark_k3y)

Digital Security

; whoami: Alexander Bolshev (@dark_k3y)

Security researcher @

Assistant Professor @ SPb ETU

Ph.D.

Distributed systems researching

Once upon a day…

… I was faced with a concrete task: evaluate the security of FDT/DTM specification in less than two month.

FDT/DTM

• “The FDT concept defines the interfaces between device-specific software components provided by the device supplier and the engineering tool of the control system manufacturer. The device-specific software component is called DTM (Device Type Manager).” © FDT Group, maintainer of FDT/DTM specification

In short:

• FDT standardizes the communication and configuration interface between all field devices and host systems

• DTM provides a unified structure for accessing device parameters, configuring and operating the devices, and diagnosing problems

FDT/DTM layers*

*Picture from http://www.automationworld.com/fdt-group-wants-your-input-yes-yours

Typical places of DTMsCorporate network ERP

MES

PLC2,3…PLC1

PLC7,8…

Field devices

AMS DCS HMI

Industrial bus

FDT/DTM simplified

Industrial bus

PAS

Modem/Gateway

CommDTM

DeviceDTM

Frame ApplicationCOM Container

CO

M C

om

po

nen

ts

Field Device

DTM multilayer concept

Looking great, right?

But…

Internals:• OLE32 & COM• ActiveX• Visual Basic 6.0• .Net• XML• JSON• Custom XML parsers• STA or RPC

Photo from http://www.flickriver.com/groups/houses_of_cards_cards_of_houses/pool/interesting/

call sub_1150F

push offset "<FDT xmlns=\"x-

schema:DTMParameterSchem"...

push offset " <DtmDevice fdt:tag=\"\">\n“

lea ecx, [ebp+var_3C]

push offset " </DtmDevice>\n“

lea ecx, [ebp+var_3C]

call sub_110D8

push offset "</FDT>\n“

lea ecx, [ebp+var_3C]

Custom XML

parsers

COM & OLE32

Internal XML

exchange

My friend’s cat simulating FDT/DTM

Research scope

• Let’s take a sample of all DTMs and find out how much of it have weaknesses and/or vulnerabilities

• Certified DTMs can be found in the catalog at http://www.fdtgroup.org/product-catalog/certified-dtms

• There are tons of DTMs

• We’ve decided to stick only to HART protocol and analyze about 100 DTMs

24 Vendors

114 DeviceDTMs

752Field

devices

from for

“Easy question”

How to fuzz 114 components in 1 month?

Why is it hard? (Problems #1)

• Different thread models & various implementations (VB 6.0, .Net, C++, …).

• Different vendors understand FDT/DTM specification differently.

• We want to fuzz DeviceDTM, how to exclude CommDTM and Frame Application?

Why is it hard? (Even more problems)

• Various licenses of DTMs (allow component to run only in specific Frame).

• HART is 1200bps really slow.

• Some DeviceDTM requires real loop with real current value.

Why is it hard? (More problems)

• Different custom (>128) HART commands for every vendor.

• DeviceDTM will send the next command only when it gets correct answer to the previous command.

• No documentation.

• Time is running up!

Last problem explained

Preamble Header Command Data Checksum

HART Packet

Managed by CommDTM

Managed by DeviceDTM

Managed by CommDTM

Commands 0-127 – described in HART specification. However, some of them

may differ from vendor to vendor.

Commands >=128 – vendor specific. Unknown packet format, (usually)

absence of any documentation.

Sent as XML binary stream from CommDTM to DeviceDTM

Problems list

Support state at fuzzing.

Increase fuzzing speed.

Fuzz unknown commands (vendor specific).

Fuzz only DeviceDTMs, exclude CommDTMs and Frame application.

Handle component license problems.

Work with components that require real CommDTMs.

Work with components that require real current loop.

Key fuzzing idea

Leave external (through CommDTM && FrameApp) fuzzing only to special cases*.

Fuzz from inside!

*when DeviceDTM has strange behaviour.

Remember the architecture

Industrial bus

PAS

Modem/Gateway

CommDTM

DeviceDTM

Frame ApplicationCOM Container

CO

M C

om

po

nen

ts

Field Device

Replace FrameApp and

CommDTMwith custom

ones

Idea sketch

Custom CommDTM(“HartFuzz DTM”)

TargetDeviceDTM

Custom Frame Application

(“FuzzFrame”)

Protocol parser

Stateful fuzzer with

modes:

• White noise

• Mutation

• Generation (“smart”)

Fuzzing?

Protocol parser

Stateful fuzzer with

modes:

• White noise

• Mutation

• Generation (“smart”)

Generating random strings of 1 to 65535 length with a bias of 1-255.

RadamsaMutationFuzzing

Photo art from http://demiart.ru/forum/image.php?i=forum/uploads/post-13689-1183931183.jpg

?? How to keep state and do smart ??fuzzing when you don’t know

packet structure

Predicting state algorithm

Receive a command request from DeviceDTM

If no command template exists do some white noise of random length

Send to DeviceDTM

Not the same command received save result as a template and send to more smart fuzzing blocks

Not so “smart” fuzzing

• I was completely lost when selecting correct fuzzer.

• Talked with Adam about Aegis but have very little time to dive into a new framework.

• Well-known HART commands cover only <20% of fuzzing area zzuf and peach would be good, but overhead.

• So I’ve decided to stick with a “silver bullet of fuzzing” – radamsa.

• Radamsa is great, but has some problems fitting in this task:• External executable (speed? scalability?)• Multiple runs required for single packet with hierarchical structure• Binary handling

• So, I’ve “forked” radamsa and created an erlang port – erlamsa

Erlamsa

• Written in Erlang.

• Recieves/sends fuzzing templates over network.

• Implements almost all radamsa mutations in the same way (excl. XML).

• Have two different XML mutators• SAX-based (complex XML structures)• Simple tag-based language

• Modified/new mutations:• Some new raw binary mutations• Length-field prediction• Binary-structure prediction• Modified ascii-bad mutator (to increase the probability of XML injections)• Some other minor corrections and new things

• Not magic or completely original tool, but just fitting “old good” radamsa in the task.

ehrtfuzz

TCP or UDP

Hart command number

XML binary stream &status

ehrtfuzz erlamsa

• Receiving commands• Keeping the state• Managing templates• White noise fuzzing and

updating templates• Simple “generation”

template-based fuzzing• Interacting with erlamsa

• Mutational-base fuzzing• Very simple packet format

detection

Excluding Frame App from fuzzing

• FuzzFrame is simple command-line based FDT Frame application for handling the DTM component.

• Initializes CommDTM and DeviceDTM, then calls “Read from device” DeviceDTM function.

• Connects back to the fuzzer via udp for controlling the current state of the component and container (and detection of non-critical bugs)

If FuzzFrame fails.

• Some DTM components have special license and could be run only from specific FDT Frame App (e.g. E+H FieldCare, VEGA PACTWare, e.t.c.).

• In this case, we could use AutoIt script for automatic call of “readfrom device” function.

• Sample script for E+H FieldCare:

Increasing the speed

• HART protocol speed is only 1200bps

• We want to exclude CommDTM and FDT Frame from fuzzing (to increase the quality of results)

• The solution: custom CommDTM and Frame application.

• So, I’ve developed HART Fuzzer DTM that emulates CommDTM.

HART FuzzerCommDTM ehrtfuzzTarget DeviceDTM

XML with command ID

ID over UDP

Command data & status

XML with binarystream & status

HART Fuzzer DTM (HrtFuzzerDTM)

Everything is logged by custom DTM!

If HrtDTMFuzzer fails…

In some cases, DeviceDTM (e.g. ABB DeviceDTMs) have “strange” thread model or require real timings/accessing the real hardware.

There are two solutions:• Emulate via Virtual COM port (we’ve used VSPE for it)

• Use ICSCorsair for emulating the device through the real current loop.

• The HRTParser Ruby gem and script is used here for working with HART on the network level (managing HART packet preamble, header and checksum).

ICSCorsair

For more info on the topic, see: “ICSCorsair: how I will PWN your ERP through 4-20mA current loop” (BH USA’14).

Worst case: real current value

• Some “very strange” DeviceDTMs works only in point-to-point HART mode and requires special HART gateway and real analog variable value through current loop.

• E.g.: if the temperature is 30 C, then the current bias in loop should be 11mA.

Analog variable (primary variable)

value.

How to control it?

I have a solution too…

• Unfortunately, I have no time to modify ICSCorsair to make it support the generation of CL analog bias.

• But I’ve found rather simple solution use AD5421 eval board for maintain current loop value.

Connecting AD5421 and ICSCorsair

To HART Gateway& power source

Loop

Problems list – finally solved!

Support state at fuzzing.

Increase fuzzing speed.

Fuzz unknown commands (vendor specific).

Fuzz only DeviceDTMs, exclude CommDTMs and Frame application.

Handle component license problems.

Work with components that require real CommDTMs.

Work with components that require real current loop.

Final architecture

FDT Frame

Target DeviceDTM

HART Fuzzer CommDTM

FuzzFrame

Vendor-specific PAS(in case of license issues)

+AutoItscript

ehrtfuzz

erlamsa

UDP

Architecture with virtual serial ports

FDT Frame

Target DeviceDTM

HART CommDTM(CodeWrights)

FuzzFrame

Vendor-specific PAS(in case of license issues)

+ AutoItscript

ehrtfuzz

erlamsa

UDPHRTParser

(Ruby)

Architecture with real loop

FDT Frame

Target DeviceDTM

HART CommDTM(CodeWrights)

FuzzFrame

Vendor-specific PAS(in case of license issues)

+ AutoItscript

ehrtfuzz

erlamsa

UDPHRTParser(Ruby)

HART “transmitter”(ICSCorsair)

USB

HART Modem

USB SPI

Current Loop

AD5421 evalboard

And finally fuzzing has begun…

Opened kingdom

Sorry, responsible disclosure

Found vulnerabilities

Vulnerable,29, 25%

Not vulnerable; 85; 75%

BY DTM

Vulnerable, 501, 67%

Not vulnerable,

251, 33%

BY DEVICE

Types of vulnerabilities

RCE 3

DoS 6

Other 9

Possible RCE 7

XML injection 2

Race Condition 2

Vendors

Some examples: XSS

“secure” memcpy

So secure!

Results && conclusions

14 vendors

29Vuls

501field

devices

from for

Good news

Recently, FDT group finally introduced a new version of FDT specification, v. 2.0. The key differences from 1.2.1 are:

• Interfaces are .Net-based

• interaction between FDT objects is based on .NET datatypes rather than XML

Bad news

FDT 2.0 isn’t a complete solution:

• Low spread over the industry

• Backward compatibility ((de)serialization to XML for working with FDT 1.2.* could cause problems)

• Managed code will not be a complete solution if unmanaged code is still used (e.g. calling old C++ code from .Net)

Tools & Links

• Tools (FuzzFrame, HrtDTMFuzzer, …) & exploits repository*

http://github.com/Darkkey/DTMResearch*the repository will be filled when >90% bugs will be fixed, currently contains AutoIt scripts, HrtDTMFuzzer component and some fuzzing scripts)

• Erlamsa repository (will be published ~ @ Feb’ 15)

http://github.com/Darkkey/erlamsa

• ehrtfuzz repository (will be published when >90% bugs will be fixed)

http://github.com/Darkkey/ehrtfuzz

• ICSCorsair repository (hardware, firmware, software):

http://github.com/Darkkey/ICSCorsair

• Old HART parser repository (ruby):

http://github.com/Darkkey/hartparser

Thanksgiving service

• Digital Security.

• Gleb Cherbov, Svetlana Cherkasova && Andrey Abakumov for participation in research, finding bugs and writing tools.

• Adam Crain really useful advices.

• Aki Helin for magnificient radamsa tool.

• George Nosenko for special binary magic and great help in reverse-engineering.

Background image from http://www.nivelco.com/site.php?upar=HOME&lang=cn

Q&A?@dsecru

@dark_k3y