Download - Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE
![Page 1: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/1.jpg)
Building Reliable, Secure and Manageable Substation
Communications
Dragan Dokic | CCIE, CISSP, MCSE
![Page 2: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/2.jpg)
Introduction - Experience
• Dragan Dokic | President, Summit Energy Tech• Focus on utility sector– Infrastructure systems management – Custom business systems software development
• 16 years of experience in IT industry• 10 years in utility sector– Managed network operations for PNGC Power [Portland,
OR] from September 2002 to October 2011– Presentation focuses on lessons learned in field network
reliability, security and manageability from this experience
![Page 3: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/3.jpg)
Introduction
• PNGC’s 2001 – 2011 field network– 92 office, substation and repeater sites at 11
distribution utilities in Oregon, Idaho• System mission– Gather real-time load data 24/7 for power
scheduling operation in Portland– Support local utility SCADA/AMI/Site Security
operations
![Page 4: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/4.jpg)
PNGC Power WAN – July 2011
![Page 5: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/5.jpg)
Toledo, OR
![Page 6: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/6.jpg)
Boardman, Oregon
![Page 7: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/7.jpg)
Junction City, Oregon
![Page 8: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/8.jpg)
Lewiston, ID
![Page 9: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/9.jpg)
Malta, ID
![Page 10: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/10.jpg)
The Moon
![Page 11: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/11.jpg)
Areas of Focus
ReliabilitySecurity
Manageability
Presentation available for download atsummitenergytech.com
in the Events section
![Page 12: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/12.jpg)
Reliability – Network Design
• Keys to success– Diversity in media
• Combine land lines, fixed wireless [private/public], mobile wireless and satellite
– Diversity in providers• Local and national
– Dynamic Routing [OSPF]• Routers exchange knowledge of local network with neighboring routers• Enterprise grade routers / switches a requirement
• Perfect world configuration– Private wired/wireless ‘island’ with two Internet gateways using
distinct media and distinct providers
![Page 13: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/13.jpg)
Connectivity overview
Primaryrouter
Backup router
![Page 14: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/14.jpg)
Link cost overview
PrimaryBackup
![Page 15: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/15.jpg)
Link cost calculationSub A -> Main Office via Satellite tunnel:3 + 1 = 4
![Page 16: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/16.jpg)
Link cost calculationSub A -> Main Office via 900Mhz+DSL tunnel:1 + 1 + 1 = 3
![Page 17: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/17.jpg)
Open Shortest PathLink cost via Satellite tunnel [4] higher than via DSL tunnel[3]; therefore, packets will traverse 900Mhz/DSL tunnel in normal operation
![Page 18: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/18.jpg)
Normal OperationOpen Shortest PathFrom substation A to Main Office
![Page 19: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/19.jpg)
Normal OperationOpen Shortest PathFrom substation B to Main Office
![Page 20: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/20.jpg)
Link down operationIf DSL tunnel is down, packets will traverse satellite tunnel;Sub A Main Office
X
![Page 21: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/21.jpg)
Link down operationIf DSL tunnel is down, packets will traverse satellite tunnel;Sub B Main Office
X
![Page 22: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/22.jpg)
Questions?
![Page 23: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/23.jpg)
Security – Overview
• Wireless link encryption• Function specific VLANs• No default routes!
![Page 24: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/24.jpg)
Wireless Link Encryption
• Media device level [e.g. Radio, Modem]– WEP, WPA, WPA2
• Routing device level [e.g. Cisco 891 router]– IPSEC
• End device level [e.g. DIGI TS4 port server]– SSL
![Page 25: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/25.jpg)
At what level to secure data?
![Page 26: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/26.jpg)
Security - Wireless Link Encryption[continued]
• Most secure option?– Use all three if management overhead is not an issue
• Most efficient but secure enough option?– Use routing device site-to-site VPN capabilities– Advantages:
• Support for best commercially available security technologies [e.g., AES-256]
• Comprehensive change logging capabilities• Standardized configuration throughout the system [less
management overhead]
![Page 27: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/27.jpg)
Security – Function Specific VLANs
• Define VLAN’s per business function– SCADA, AMI, Security System, Wireless, VOIP, Network Mgmt.
• Firewall traffic between VLANs on need-to-access basis– E.g., Prevent personnel attached to substation wireless VLAN to
access documentation stored on a server at the main office from accessing recloser controls in the SCADA VLAN
• Reliability advantages– Non-critical VLANs [e.g. AMI, security] can be shut down
automatically/remotely if link quality is too poor to carry all traffic, but good enough to carry SCADA
![Page 28: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/28.jpg)
One VLAN per business function
![Page 29: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/29.jpg)
High-speed link outage scenario
![Page 30: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/30.jpg)
Security – No Default Route!
• Do not use default routes through service provider-supplied gateways
• Define a single host route back to the main office, then establish default route through VPN tunnel
• This is the most effective method to prevent attacks sourced from the Internet
• Always use in conjunction to regular firewall configuration lists [not a substitute!]
![Page 31: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/31.jpg)
Less secure
Provider gateway
![Page 32: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/32.jpg)
More secure
Provider gateway
![Page 33: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/33.jpg)
Questions?
![Page 34: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/34.jpg)
Manageability - Overview
• Tools – network management systems• Addressing – developing a scheme• Watchdog system – preventing lockout
![Page 35: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/35.jpg)
Manageability – Tools
• Network Management Systems [NMS]• Protocols used
• SNMP, Syslog, ICMP, HTTP
• Applications• PRTG• Solarwinds Syslog
![Page 36: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/36.jpg)
Manageability – Tools [continued]
• How to collect data? Push vs. Pull– Pull: Poll devices using SNMP/HTTP/ICMP at regular intervals
[e.g., every – Push: Devices send data per defined event triggers
– SNMP traps– Syslog messages
• What data to collect?– Availability [ping]– Network utilization– Input voltages– RSSI [radio link quality]
![Page 37: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/37.jpg)
Manageability – Tools [continued]
• Pull example: – 5 minute SNMP poll of UPS for input voltage– If voltage drops below threshold of 108VAC for a duration of
time longer than 5 minutes, an alert will be triggered by NMS [e-mail, text message, event log]
– But what if voltage drops for 2 minutes only in between polls? You may not know it even happened.
• Push comes to rescue:– UPS sends SNMP trap to NMS as soon as voltage drops below
108VAC– Alert is triggered by NMS when trap is received
![Page 38: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/38.jpg)
Paessler PRTG – Screen shot
![Page 39: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/39.jpg)
Solarwinds Kiwi Syslog – Screen shot
![Page 40: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/40.jpg)
Manageability – Addressing
• Develop consistent scheme to use system wide• Recommended private range: 10.0.0.0/8– First octet: same for entire system– Second octet: site ID [e.g. 8=Springfield Sub]– Third octet: business function ID [e.g., 4=AMI]– Fourth octet: device itself [e.g., Collector #1]
1st octet ‘fixed’
2nd octet = site ID 3rd octet = vlan/business function
4th octet = device
Subnet Mask [255.255.255.0]
![Page 41: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/41.jpg)
Manageability – Addressing [continued]
• Large network?– Group sites by region using second octet– Allows for address summarization if needed.
• Example:– Eastern division region:
• 10.64-127.0.0 • Summary address: 10.64.0.0/10
– Western division region:• 10.128-191.0.0 • Summary address: 10.128.0.0/10
![Page 42: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/42.jpg)
Manageability – Watchdog System
• General concept– Reboot key remote communications devices if
connectivity to central site is interrupted• Benefit– Prevent unnecessary site visits due to• Operator error• Device lock-up [e.g., buggy firmware, heat issues]
![Page 43: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/43.jpg)
Manageability – Watchdog System [continued]
• Hardware requirements:– SNMP-capable switched PDU with task scheduling
and delayed power cycling command capabilities– Example: APC AP7900 8-port 15A PDU
• Software capability requirements:– Centralized command override mechanism using
NMS– Send SNMP ‘Set’ to cancel pending power cycling
command
![Page 44: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/44.jpg)
Manageability – Watchdog System Example
• ‘Delayed’ power cycle schedule is defined on PDU:– Outlets to power cycle: 1,2 [e.g., radio, router] – Frequency: 60 minutes– Command execute delay:30 minutes
• Network management system running at main office sends an SNMP delayed power-cycle command cancel message– Frequency: every 5 minutes
• Process– If delayed power cycle cancel command cannot reach the PDU at least
one time during the 30 minute reboot delay period, outlets 1 and 2 will be power cycled and communication will (hopefully!) be restored
![Page 45: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/45.jpg)
Questions?
![Page 46: Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE](https://reader030.vdocuments.site/reader030/viewer/2022032517/56649ca45503460f94964af4/html5/thumbnails/46.jpg)
Thank you!