Building Reliable, Secure and Manageable Substation

Communications

Dragan Dokic | CCIE, CISSP, MCSE

Introduction - Experience

• Dragan Dokic | President, Summit Energy Tech• Focus on utility sector– Infrastructure systems management – Custom business systems software development

• 16 years of experience in IT industry• 10 years in utility sector– Managed network operations for PNGC Power [Portland,

OR] from September 2002 to October 2011– Presentation focuses on lessons learned in field network

reliability, security and manageability from this experience

Introduction

• PNGC’s 2001 – 2011 field network– 92 office, substation and repeater sites at 11

distribution utilities in Oregon, Idaho• System mission– Gather real-time load data 24/7 for power

scheduling operation in Portland– Support local utility SCADA/AMI/Site Security

operations

PNGC Power WAN – July 2011

Toledo, OR

Boardman, Oregon

Junction City, Oregon

Lewiston, ID

Malta, ID

The Moon

Areas of Focus

ReliabilitySecurity

Manageability

Presentation available for download atsummitenergytech.com

in the Events section

Reliability – Network Design

• Keys to success– Diversity in media

• Combine land lines, fixed wireless [private/public], mobile wireless and satellite

– Diversity in providers• Local and national

– Dynamic Routing [OSPF]• Routers exchange knowledge of local network with neighboring routers• Enterprise grade routers / switches a requirement

• Perfect world configuration– Private wired/wireless ‘island’ with two Internet gateways using

distinct media and distinct providers

Connectivity overview

Primaryrouter

Backup router

Link cost overview

PrimaryBackup

Link cost calculationSub A -> Main Office via Satellite tunnel:3 + 1 = 4

Link cost calculationSub A -> Main Office via 900Mhz+DSL tunnel:1 + 1 + 1 = 3

Open Shortest PathLink cost via Satellite tunnel [4] higher than via DSL tunnel[3]; therefore, packets will traverse 900Mhz/DSL tunnel in normal operation

Normal OperationOpen Shortest PathFrom substation A to Main Office

Normal OperationOpen Shortest PathFrom substation B to Main Office

Link down operationIf DSL tunnel is down, packets will traverse satellite tunnel;Sub A Main Office

X

Link down operationIf DSL tunnel is down, packets will traverse satellite tunnel;Sub B Main Office

X

Questions?

Security – Overview

• Wireless link encryption• Function specific VLANs• No default routes!

Wireless Link Encryption

• Media device level [e.g. Radio, Modem]– WEP, WPA, WPA2

• Routing device level [e.g. Cisco 891 router]– IPSEC

• End device level [e.g. DIGI TS4 port server]– SSL

At what level to secure data?

Security - Wireless Link Encryption[continued]

• Most secure option?– Use all three if management overhead is not an issue

• Most efficient but secure enough option?– Use routing device site-to-site VPN capabilities– Advantages:

• Support for best commercially available security technologies [e.g., AES-256]

• Comprehensive change logging capabilities• Standardized configuration throughout the system [less

management overhead]

Security – Function Specific VLANs

• Define VLAN’s per business function– SCADA, AMI, Security System, Wireless, VOIP, Network Mgmt.

• Firewall traffic between VLANs on need-to-access basis– E.g., Prevent personnel attached to substation wireless VLAN to

access documentation stored on a server at the main office from accessing recloser controls in the SCADA VLAN

• Reliability advantages– Non-critical VLANs [e.g. AMI, security] can be shut down

automatically/remotely if link quality is too poor to carry all traffic, but good enough to carry SCADA

One VLAN per business function

High-speed link outage scenario

Security – No Default Route!

• Do not use default routes through service provider-supplied gateways

• Define a single host route back to the main office, then establish default route through VPN tunnel

• This is the most effective method to prevent attacks sourced from the Internet

• Always use in conjunction to regular firewall configuration lists [not a substitute!]

Less secure

Provider gateway

More secure

Provider gateway

Questions?

Manageability - Overview

• Tools – network management systems• Addressing – developing a scheme• Watchdog system – preventing lockout

Manageability – Tools

• Network Management Systems [NMS]• Protocols used

• SNMP, Syslog, ICMP, HTTP

• Applications• PRTG• Solarwinds Syslog

Manageability – Tools [continued]

• How to collect data? Push vs. Pull– Pull: Poll devices using SNMP/HTTP/ICMP at regular intervals

[e.g., every – Push: Devices send data per defined event triggers

– SNMP traps– Syslog messages

• What data to collect?– Availability [ping]– Network utilization– Input voltages– RSSI [radio link quality]

Manageability – Tools [continued]

• Pull example: – 5 minute SNMP poll of UPS for input voltage– If voltage drops below threshold of 108VAC for a duration of

time longer than 5 minutes, an alert will be triggered by NMS [e-mail, text message, event log]

– But what if voltage drops for 2 minutes only in between polls? You may not know it even happened.

• Push comes to rescue:– UPS sends SNMP trap to NMS as soon as voltage drops below

108VAC– Alert is triggered by NMS when trap is received

Paessler PRTG – Screen shot

Solarwinds Kiwi Syslog – Screen shot

Manageability – Addressing

• Develop consistent scheme to use system wide• Recommended private range: 10.0.0.0/8– First octet: same for entire system– Second octet: site ID [e.g. 8=Springfield Sub]– Third octet: business function ID [e.g., 4=AMI]– Fourth octet: device itself [e.g., Collector #1]

1st octet ‘fixed’

2nd octet = site ID 3rd octet = vlan/business function

4th octet = device

Subnet Mask [255.255.255.0]

Manageability – Addressing [continued]

• Large network?– Group sites by region using second octet– Allows for address summarization if needed.

• Example:– Eastern division region:

• 10.64-127.0.0 • Summary address: 10.64.0.0/10

– Western division region:• 10.128-191.0.0 • Summary address: 10.128.0.0/10

Manageability – Watchdog System

• General concept– Reboot key remote communications devices if

connectivity to central site is interrupted• Benefit– Prevent unnecessary site visits due to• Operator error• Device lock-up [e.g., buggy firmware, heat issues]

Manageability – Watchdog System [continued]

• Hardware requirements:– SNMP-capable switched PDU with task scheduling

and delayed power cycling command capabilities– Example: APC AP7900 8-port 15A PDU

• Software capability requirements:– Centralized command override mechanism using

NMS– Send SNMP ‘Set’ to cancel pending power cycling

command

Manageability – Watchdog System Example

• ‘Delayed’ power cycle schedule is defined on PDU:– Outlets to power cycle: 1,2 [e.g., radio, router] – Frequency: 60 minutes– Command execute delay:30 minutes

• Network management system running at main office sends an SNMP delayed power-cycle command cancel message– Frequency: every 5 minutes

• Process– If delayed power cycle cancel command cannot reach the PDU at least

one time during the 30 minute reboot delay period, outlets 1 and 2 will be power cycled and communication will (hopefully!) be restored

Questions?

Thank you!