practical communications considerations for protection engineers
TRANSCRIPT
1
Practical Communications Considerations for Protection Engineers As Submitted to the 2013 Georgia Tech Protective Relay Conference
Adrian G Zvarych, PE
Communications Systems Engineering
Power Grid Engineering, LLC
Winter Springs FL
Iza Pomales
P&C Engineering
Power Grid Engineering, LLC
Winter Springs FL
Jose Rodriguez
Director of Engineering
Power Grid Engineering, LLC
Winter Springs FL
Dolly Villasmil
P&C Engineering
Power Grid Engineering, LLC
Winter Springs FL
Abstract - Very few devices are currently installed in
substations without some form of communications connection.
There is a clear trend toward establishing data connectivity via
Ethernet due to generally higher data rates, and cost effective
connections. Whether an application is for Supervisory Control
and Data Acquisition (SCADA), line relaying, remote
engineering access or Synchrophasors, the Intelligent Electronic
Device (IED) is manufactured featuring a variety of
communication ports including RS-232, RS-485, Ethernet, and
fiber that are connected to, and communicating with at least one
other remote device. Protection engineers typically have a
limited role in communication applications, thus they may not
have a full understanding of a communication networks’
capabilities.
This paper is intended to improve the understanding of the
commonly used forms of communication connections used at a
substation. This paper discusses many communication services
found within a substation.. Each area within itself is worthy of a
text books’ worth of attention. The locus of this document is to
extract highlights of each circuit or application type as pertinent
to a protection engineer for the purpose of gaining a better
understanding of the type of circuit and some of the key roles the
application has in a substation environment.
Basic network design considerations and technologies will also be
reviewed to ensure the network is able to support the
performance criteria imposed by the circuit type. Additionally,
this paper provides design considerations for the different forms
of connections mentioned above. Lastly, it this paper provides
guidance on how these circuit types are applied in substation
applications.
After reviewing this paper or attending the presentation, the
relay engineer will have a broader understanding of the different
circuit types, how they are used, understand the cost and benefits
unique to each circuit type, and be able to communicate those
needs effectively with a Telecom or Information Technology
professional.
Keywords - Substation; Communications; Serial; Ethernet;
SCADA, RS-232, RS-485, IED, Fiber, Protection, Network
Design; IT, Telecom.
I. COMMUNICATION NETWORK TYPES
A. Introduction
Communication circuit connections for a particular function
such as SCADA, line relaying, metering, etc., are typically
pre-defined by industry standards and somewhat dependent on
what type of communication interface a manufacturer offers.
Examples of interfaces are RS-232 or RS-485 for SCADA,
IEEE-C37.94 for line current differential relaying, etc. In
transporting the circuits from one IED to another, there are a
growing range of options and criteria to consider.
Connections between sites must typically be developed and
coordinated with other teams within an organization such as
IT and Telecom groups. The Protection & Control and
SCADA engineer must establish performance criteria for each
type of circuit required at a substation or plant site, especially
since it is increasingly likely that a packet-based network will
be transporting the data as opposed to legacy time-domain
based networks. The Information Technology and
Telecommunications engineers can use this information to
properly develop a secure and reliable information delivery
network to deliver the information in an appropriate manner.
The performance criteria should include at least the following
parameters:
Application (Line relaying, SCADA, metering, etc.)
Connection type (RS-232, RS-485, Ethernet, etc.)
Anticipated bandwidth needs (Mbps or kbps)
Maximum tolerable latency
Maximum tolerable outage time during network
switching events
Maximum tolerable asymmetrical delay
The need for the circuit to remain enabled before,
during, and after a power system fault
Maximum circuit restoration time after a
communications outage
2
Note that some of the above parameters are dependent on an
IED’s design constraints. Other parameters may be more
dependent on power system or other operational constraints.
Another consideration for circuit reliability criteria involves
closely coordinating design and maintenance practices with
the team that is designing and maintaining the telecom
network.
As an example, a telecom team may consider replacing a 48V
battery string that is supplying backup power to network
equipment carrying line relay and SCADA circuits without
any spare battery in place, to provide backup during the
replacement. In contrast, a crew replacing the substation
battery might typically connect a temporary mobile battery to
the station’s 125V charger as a backup source during a
substation battery replacement task. In certain cases, having
no backup even during a telecom battery replacement may be
unacceptable. The Protection & Control engineer must ensure
a holistic approach to overall reliability if a circuit is
considered, including telecom maintenance processes and
procedures.
B. Leased Services
Leased Services may be required when utility-owned
communication paths between sites is not available. A typical
remote location could be:
Power Plant sites (including administration
buildings)
Substations
“Outside the fence” equipment such as reclosers,
capacitor banks, voltage regulators
A site with the presence of utility-owned broadband
network equipment for backhauling communications
traffic
A control center site, where SCADA or other data
circuits from substations or other field sites terminate
An adjacent utility or operating entity requiring
SCADA, relaying, metering, or other communication
circuits
Some commonly ordered leased circuit options available for
analog and digital circuits include:
Plain Old Telephone Service (POTS). A
conventional dial-up phone line, registered with the
E-911 dispatch center
Four-wire AC Data. This can provide audio tone
relaying and legacy SCADA connections. Ordered
and provisioned between two sites
Four-wire digital DDS. Used for a 64/56 kbps RS-
232 SCADA type circuit or low-bandwidth packet
based Ethernet over a Time Division Multiplexed
(TDM) network. Ordered and provisioned between
two sites [1]
Frame Relay. Used for SCADA, an early form of
packet-based connectivity with a tightly defined
network boundary. A utility may own and manage
part of a Frame Relay Cloud [2]
DS-1 with 1.544Mbps bandwidth, Can be used for
extending Ethernet or 56/64 kbps DS-0 channels or
both, between sites. These circuits are ordered and
provisioned between two points
Cellular
Selecting the circuit type is dependent on the amount of data
required to be transferred and the interfacing capability at the
substation IED and a utility’s practices.
For SCADA circuits, the SCADA Master controls all data
transmission by polling equipment at each substation, keeping
substations from interrupting each other. Analog circuits are
limited to 33 kbps (or lower) by the physics of the digital-to-
analog conversion. For digital circuits, data can be sent at a
rate up to 1.5 Mbps, known as a DS-1 rate in Time Domain
Multiplexing (TDM) networks. Equipment needed to build
the network for digital circuits are more sophisticated and are
more expensive than designing analog circuits. However,
industry trends indicate moving away from individual two-
and four-wire serial circuits and toward packet-based
networks. Even Frame Relay networks are being replaced by
a type of network known as Multi Protocol Label Switched
(MPLS) networks [3].
Any leased service generally incurs a monthly fee. The rate
structure is typically based on a combination of circuit type
and physical distance from one end point to the other. For
communication circuits leased over a larger geographic area, it
is not uncommon for multiple communication carriers to hand
off the circuit between each other, raising the cost, potentially
negatively affecting reliability and restoration times.
Advantages of using leased services to establish
communication include avoidance of large capital expenses to
build utility-owned infrastructure, small initial capital outlay,
network maintenance is performed by others, and the expense
of changing and transporting different circuit types is incurred
by others. Some disadvantages are repair and maintenance
are not controlled by the utility, circuits may not be available
at some sites, metallic links require protection against Ground
Potential Rise (GPR), and recurring Operating Expense
(OPEX) costs. Total cost of ownership must include life cycle
costs including the avoidance or addition of training and
staffing costs.
When selecting a leased services option, it is important to
keep in mind the criticality of protecting communication
facilities entering an electric substation. The basic objectives
for protecting communication facilities entering a power
substation are to ensure personnel safety, protect the
3
telecommunications site and terminal equipment, and maintain
reliability of service. This is defined by IEEE Standard 487-
2007, which states that High Voltage Protection (HVP) is
required at sites with a peak Ground Potential Rise (GPR)
greater than 1,000 volts. It also states that for sites with a GPR
of 1,000 volts peak or less, gas tubes or other shunting devices
are suitable.
A notable industry trend is that leased services providers are
increasingly likely to provide communication services with
fiber optic access points as opposed to copper circuits. This
type of connection can provide much higher bandwidth and
theoretically better performance in terms of mitigating the
effects of Ground Potential Rise (GPR) to which copper
communication circuits are subject.
1) Outside Plant Facilities
In order to move data between sites, outside plant facilities are
required. The facilities are typically some type of cable as
opposed to wireless forms of connection.
As the cost of copper and other conductive metals continues to
rise, the cost of optic cables is continuing to fall. See the
figures below for the trends.
Figure 1.1
http://www.infomine.com/investment/metal-prices/copper/all/
Cost of Copper, January 1989 – February 2013
Materials costs have shifted such that the cost of a
multiconductor copper cable used for RS-232 signaling in a
control house can be more expensive than a multimode fiber
patch cable of the same length. The design engineer should
consider the cost of IEDs equipped with optical interfaces,
which can add back to any cabling savings. Still, the
immunity to electrical noise which fiber has can often justify
the incremental added costs of fiber connections as opposed to
copper connections.
Fiber cables are the most dominant type of communication
cable installed today by electric utilities to provide
connectivity between sites. More considerations will be
discussed later in this paper. Fiber cables are constructed
either overhead or underground, each having certain
reliability, design, construction, and cost challenges. Facilities
can be owned and operated by either the electric utility or by a
leased services provider.
2) Pilot Wire Systems
Some utilities may own and operate pilot wire systems [4].
The connecting media between devices consists of pairs of
metallic conductors housed in a reinforced, armored cable and
is typically installed underground. The primary function of a
pilot wire system is to transport protective relay signals
between substation sites in a point-to-point fashion. Some
utilities may also use the copper pilot wire cables to transport
SCADA, metering, telephony, or other data between sites.
Pilot wire systems using metallic interfaces are rapidly
becoming obsolete, are expensive to construct and maintain,
can take longer to restore than fiber, are subject to the effects
of earth currents due to imbalance currents, lightning
discharges, and earth currents responding to solar flares. These
schemes are being replaced by fiber based connections.
Any copper cable entering a substation’s “Zone of Influence”
as defined in IEEE Standard-80 and -487, requires special
protection in order to mitigate the effects of earth currents as a
result of load imbalance or earth faults. Types of equipment
include isolation transformers and neutralizing reactors, which
can be expensive and difficult to have installed properly.
Bandwidth limitations and rapidly increasing maintenance and
cable replacement costs have driven many utilities to replace
aging pilot wire systems with optically based network
equipment to manage protective relaying, SCADA, telephony,
and other growing communication needs in today’s
substations.
The conversion from any point-to-point copper based network
to an optical ring-based network requires through-node delay
and cross-ring bandwidth management, specifically for
differential relay circuits. The protection engineer must
closely work with the telecom design engineer to arrive at a
cost effective solution that will satisfy relay system
performance needs.
C. Microwave
The term “Microwave” is somewhat loosely used to define the
electromagnetic frequency range between about 1-300 GHz,
although some text books define the microwave range as
beginning at around 300MHz. For the purpose of this
document, values above 300MHz will be included in the
family of microwave frequencies and generally discussed in
this section.
Power Line Carrier (PLC) systems use a range of frequencies
in the 30 – 450kHz range. PLC is still a viable and perhaps
arguably expensive option in many line protection
applications. It typically offers no other advantages or
4
functions to any other communication need in a substation and
is dedicated primarily to protection applications.
Figure 1.2
Electromagnetic Frequency Spectra
In general, microwave frequencies in an electric utility have
historically been applied as shown in the following chart:
Frequency Application
400MHz Legacy vehicle communications
700MHz Emergency communications
800MHz Handheld two-way radio
900MHz SCADA
2.5GHz WiMAx applications – Smart Grid
2 – 22GHz Legacy digital Point-to-Point microwave
The higher the frequency, generally speaking, the more
bandwidth can be transported but the shorter the reach. A
22GHz microwave system may have a reach of less than eight
miles depending on a variety of design and environmental
factors. A 900MHz system can reach to about 25 miles
reliably. Although typically not used in the electric utility
industry, transceivers in the shortwave frequency range of 1.8
– 30MHz, distances of over 5,000 miles are possible.
Microwave systems considered as being able to transport
Wide Area Network/Broadband traffic will generally operate
in the 12 – 22GHz range. These microwave devices may have
optical interfaces that enable them to bridge up to an OC-3
(about 150 Mbps of bandwidth) across an area that has no
fiber connectivity. Legacy serial SCADA traffic can be
transported 900MHz radio, with transport capacities of up to
three individual DS-0 circuits. Secure IP radios are emerging
in our industry, also within the 900MHz range.
1) Backhaul Microwave Networks
A backhaul microwave network is designed to move a
relatively large amount of communications traffic between
sites. Transporting more than 10 Mbps worth of traffic
between sites generally qualifies the network as being a
broadband network. Backhauling data involves moving data
from one or two core sites to and from mulltiple remote sites,
akin to a typical SCADA communications model. A backhaul
network can operate at one of many frequencies that the
Federal Communications Commission (FCC) manages.
Practically speaking, bandwidth available on a radio-
frequency based network is limited to about 150 Mbps, or an
OC-3’s worth of bandwidth in Synchronous Optical NETwork
(SONET) terms. Smart Grid initiatives have led to the utilities
developing backhaul networks in packet-based WiFi networks
(defined in the IEEE 802.11 family of standards) and WiMAX
(as defined in the IEEE 802.16 standards) [5] [6].
2) 900MHz Microwave Systems
Two categories of commercial spectrum exist in the United
States, licensed and unlicensed. Licenses from the Federal
Communication Commission (FCC) allow companies to have
exclusive access to particular frequencies within a
geographical area. Since licensed spectrums in the 900MHz
range can be limited to several hundred kHz of bandwidth,
they are difficult and expensive to obtain in large blocks. On
the other hand, unlicensed spectrum is available for
commercial use at no cost. Both licensed and unlicensed 900
MHz spectrum is commonly used for distances up to about 23
miles, depending on conditions [7].
900MHz microwave systems are widely applied in SCADA
systems primarily because of their accessibility, availability,
relatively low cost, and potentially license free status. This
frequency range tolerates interference, extreme weather
conditions, and operates in a data only band. It is mostly used
by utilities and other commercial and industrial businesses and
not by the general public.
One point to realize is that when referring to a “900 MHz”
application, the actual frequencies assigned or used lie within
the 900 – 928 MHz range.
From a SCADA application perspective, the power industry
has debated the issue of utilizing licensed or unlicensed
900MHz systems for many years. Subjects ranging from
security, interference, cost, modernization of assets including
mobile data infrastructure have been taking center stage in
recent times within the power industry.
Within the 900MHz band, there are shared unlicensed and
licensed spectra. 900MHz devices can be operated in either
point-to-point or point-to-multipoint fashion. Obtaining an
FCC license for a path guarantees the subscriber exclusive use
to that frequency for the end points included in the original
license application. An operator occupying the same
frequency range and causing interference on the licensed
network would be investigated by the FCC. Licensed radio
systems operate at fixed frequencies as opposed to 900MHz
5
Spread Spectrum systems, which employ frequency hopping
schemes within a specific frequency range to mitigate
interference [8].
Due to the limited space between licensed spectrums,
powerful and noisy transmitters can interfere with the
channels adjacent to those licensed by smart grid vendors or
others. Licensed spectrums are not necessarily more secure
than unlicensed ones. Strong security relies on
communication protection implementation such as applying
encryption algorithms to the channel or channels.
The total cost of a wireless system must consider a wide range
of factors. Some of these include tower, site prep, standby
power systems, Federal Aviation Administration (FAA)
required obstruction lighting and monitoring systems,
coverage gaps, spectrum costs, and the increase of overall
total cost of ownership due to maintenance.
Developed roughly 50 years ago by the American military,
Spread Spectrum is a technology whose operation essentially
is to spread data across a wide ‘hopping’ frequency band and
minimize the effect of interference to the transmission of
signals. Spread Spectrum is considered a class of unlicensed
equipment.
900MHz Spread Spectrum
• No Federal Communications Commission (FCC)
licensing required
• Multi-protocol traffic supported
• Spread Spectrum not on dedicated frequency but
offers immunity to interference
• Broadly used for SCADA
• Unlicensed product operating with a mixture of other
products and applications (oil, gas, wastewater,
railroads, etc.)
• No guarantee on data integrity
• Range of operation is up to 23 miles, depending on
conditions
• Point-To-Point or Point-To-Multipoint capable
900MHz Licensed
• Dedicated frequency, license issued by FCC
• Long lead time for obtaining license
• Initial and annual renewal fees
Microwave Systems (Above 6GHz)
• All licensed Frequency Assignments available
• More flexible waveguide and antenna requirements
• Private Network
• Point-to-Point connections
• Can support OC-1 or OC-3 ‘missing link’ fiber
networks
• Data rates up to 150 Mbps
• Simpler installation than cable technology
• Restrictions are higher on dedicated paths between
stations
• Low digital data bit rate
• Frequency assignments often unavailable in urban
areas
• Distance covered decreases with increased frequency
In general, licensed systems offer the protection from
interference by the FCC, while Spread Spectrum systems are
subject to interference from other Spread Spectrum systems
installed nearby with the same hopping patterns. Also, it is
impossible to guarantee that obstructions will not be built
within any RF path’s line-of-sight or within the Fresnel effect
zone around the line-of-sight that might affect performance
after installation.
D. Satellite
Satellite communication has been used in the utility industry
for quite some time and continues to be one of the fastest
growing technologies. Connectivity to remote sites can be
easily accommodated by satellite services. With today’s
increasing demand for power, utilities must invest in and
provide a point of balance within energy improvements and
business operations.
Inside the energy improvement category, communications
along with its specific components play a key role. There is a
widespread misconception that satellite communications
comes along with staggering costs in addition to fallible
accessibility and reliability for critical utility applications.
Reality is that satellite communications technology has been
developing to become two-way, providing state of the art
broadband connectivity. Satellites are high-speed
communication systems built on IP integrated with basic
communications technology. In addition, Satellites’ IP
platform extends to remote locations, in other words, they are
about to reach large geographical areas in a cost effective
manner.
Technological advances have allowed for the development of
Very Small Aperture Terminal (VSAT) systems. These allow
for a user platform to provide end-to-end communication
between devices. An External protocol is required to properly
decode the message being transmitted through the VSAT
system. The network protocol utilized by the VSAT facilitates
the efficient transfer of user data over the satellite link. This
VSAT technology allows for the use of a much smaller
antenna, therefore decreasing capital costs significantly.
A study performed by UTC in April 2011 called, “Strategic
Assessment of Satellite Usage in the Utility Industry,” found
that the greatest benefit to utilities from satellite technology is
that it enables ubiquitous network connectivity across the
utility’s service areas. According to the research, “satellite’s
6
portability – its ability to bring communications connectivity
where needed – is a related key benefit. Around 60 percent of
utilities use some form of satellite communications today and
about one-fifth of utilities that are not currently using satellite
communications technology, plan to do so within the next two
years [9] [10].
Overall, satellite communications has been attractive to
utilities because of the extensive range capabilities, low error
rates, and easy access to remote sites amongst its primary
benefits. The design engineer must carefully balance the trend
of growing needs for data to be extracted from field devices
and a satellite system’s relatively low bandwidth.
E. Fiber
Fiber optic cable provides the highest bandwidth of any other
form of communication connection. Most recent industry
trends indicate that the amount of bandwidth transported over
fiber on the Internet tends to double every nine months [11].
It is also the most immune communication media type to any
RFI and electric system transients. Depending on utility
practice, fiber optic cable can be installed underground or
overhead.
Overhead fiber optic cable falls into three basic types:
All Dielectric Self Supporting (ADSS)
Optical Ground Wire (OPGW)
Access Wrap (aka “Lashed”)
Lashed fiber optic cables do not have a built in strength
member to support its own weight as installed aerially, and
must be attached to or “lashed” on to a high strength steel
cable that is designed to be wrapped around the cable carrying
the fibers and attached at each utility pole. This type of
installation is preferred by telecom and cable providers.
Electric utilities generally prefer to install ADSS fiber cable
on existing structures as underbuilt retrofits, in either the
“Communications” or the “Supply” space as defined by the
National Electric Safety Code. Optical Ground Wire (OPGW)
cable is typically installed in the overhead shield position on a
transmission line. OPGW installations are more common as
initial installations on new line construction, or where line
conductors or the shield wire are being replaced.
Figure 1.3
ADSS Cable, showing multiple buffer tubes with glass
multiple strands and central strength member
Image courtesy of AFL
Figure 1.4
OPGW Cable, showing single aluminum buffer tube with
multiple fiber-containing buffer tubes Image courtesy of AFL
Fiber cable designed for lashing are similar to the ADSS
design with the exception that a strength member is not
required to be as robust, since an external means will be used
to provide supporting strength for installation.
The cost of the glass is negligible as compared with the cost of
the mechanically supporting and separation portions of the
cable, making high-count fiber cables much more commonly
installed. The balance point for overhead installations is that
as the diameter of the cable increases, so does wind loading.
There are cases where the structures supporting the fiber cable
may not be able to handle horizontal wind loading on fiber
7
cables without modification or replacement. The design
engineer must consider not only wind loading but also the
number of fibers required to achieve the desired network
capacity.
Most services at a substation can be met by just a single pair
of fibers in a multiplexed or packet-based network. Other
fibers within the cable can be used to support non-substation
corporate applications or leased out to other companies to
generate revenue, or dark fibers swapped with other
companies to provide redundancy on the network for one or
both companies.
Multimode fiber (MMF) refers to the different modes the light
rays travel down the optical fiber. For instance, for each pulse
of light delivered through the optical fiber, the light travels
through the fiber core along multiple paths. This behavior
results in the pulses of light spreading out thus limiting the
bandwidth MMF can support being utilized mostly on short
distance applications. The most common types of multimode
fibers used in the utility industry are in the cladding
dimensions of 62.5/125 µm and 50/125 µm. The large core
size of multimode fiber cable allows for inexpensive
connectivity, greater durability and low-cost light source
capability. Due to these characteristics, they are used for data
communications links with local area network (LAN), more
specifically, short distance LAN applications less than
approximately 6500ft between connections points. 50um core
multimode fiber was developed for the emerging giga-bit
Ethernet networks.
Today’s trend in terms of fiber usage is toward single mode
fiber. The cost for single mode optics has decreased, research
continues to enable increasing bandwidth on singlemode glass,
and long distances are able to be traversed.
Single mode fiber (SMF) has a much smaller light-carrying
region, approximately 7.2 - 8.3 µm in diameter. It has a very
large information carrying characteristics and minimal loss
properties as compared to multimode fiber. SMF allows a
single path only for each pulse of light to carry through the
core of the fiber. Single mode light can deliver information at
distances up to 160 km. This type of application combines the
use of high precision laser-based transceivers for the design of
networks capable of transmitting voice & data messages over
100 Gbps for long distances.
Figure 1.5
Multimode and Singlemode Fiber
Some of the typical applications perfectly suitable for SMF
but not limited to are: wide area network (WAN),
metropolitan area networks (MAN), coarse wavelength
division multiplexing (CWDM) and dense wavelength
division multiplexing (DWDM). DWDM technologies enable
many multiple wavelengths of light to be combined on a
single fiber.
The use of optical transmitters is essential for fiber optics
applications and can be in the form of light emitting diodes
(LEDs) or laser diodes. Optical transceivers typically operate
at 850, 1310 or 1550 nm depending on the application. 1550
nm range optics are considered ‘extended’ reach and while
more expensive than 1310nm class optics, can span distances
up to 160 km depending on the application and bandwidth.
850 nm optics are applied for short-reach, campus type
multimode fiber applications.
Entities that transport bulk quantities of data from region to
region may employ Dense Wave Division Multiplexing. This
technology enables multiples of OC-48 or the equivalent
Ethernet bandwidth, to be light-superpositioned on one or two
fibers. In order to accomplish this, SONET or Ethernet node
optics must be converted to operate within slightly different
wavelengths of light, then combined through a precise prism
to inject the light onto the glass fiber. At the receive end, each
receiver is precisely tuned to only receive the frequency of
light out of the entire spectrum that has been injected by its
corresponding transmitter at the remote end. This equipment
tends to be costly and generally is not applied at the substation
level.
Utilities have recently been applying OC-48/2.5 Giga-
bit/second fiber networks at the substation to transport both
serial and Ethernet traffic. As Ethernet makes a stronger
presence in the industry and technology advances, these
networks may be replaced by 10 Giga-bit/second or higher
bandwidth in years to come. The trend will be driven by rapid
8
migration to packet-based networks, and growing needs for
more data to be retrieved.
The principal advantages of using overhead construction to for
fiber cables are: security, constructability, and cost.
Security is given by the fact that these networks are located
near energized conductors, which make it less attractive for
thieves to reach; constructability in terms of infrastructure
already available for the installation of these overhead
conductors which eases the efforts of installing fiber optic
cables; and lower cost for overhead networks since initial cost
is reduced by the existence of physical structures.
Fiber optic cable is available in several styles, depending on
where the cable is installed. All-Dielectric Self-Supporting
(ADSS), Optical Ground Wire (OPGW), and AccessWrapTM
(aka “Lashed”) are the three dominant cable types used to
connect substations together in a network. Their
characteristics are shown in the table below.
Type Characteristics
ADSS
Cable physically strong to be
able to support itself between
poles, and any additional
weight imposed from weather
conditions
Independent from electrical
cables even though they share
same poles
Design of fiber network using
ADSS is dependent on the
landscape and can limit the
application using this type of
cable
OPGW
Replaces shield wire electrical
conductor that has optical fibers
built into it
Replaces shield wire on
electrical overhead lines and
does not affect the mechanical
or electrical rating of the line
Most secure and delicate of all
fiber optics cable types since it
physically interacts with
electrically energized
conductors and it directly
impacts operation of both
networks
Usually part of new
construction
AccessWrap
Adds fiber optic cable by
wrapping it securely on power
conductors and therefore its
own weight does not impact the
strain of the cable
Installation is done by using a
special device that travels
through the host conductor and
clamps are used on each side of
the cable to hold it on the poles
It does not pose any additional
load on the supporting power
conductors and it does not
reduce the clearance distance
under the line
Maintenance of the fiber optics
cable can be accomplished
without disturbing the power
conductor
Table 1.1
Characteristics of Fiber Types
Structural analysis of the poles onto which fiber cables are
retrofit must be done in order to assure the structure can
support the additional wind and ice loads. Analysis is
typically performed by engineers on the distribution or
transmission teams.
Underground fiber cables are designed for high pulling tension
and lubricants to reduce friction during installation. As during
the pulling of overhead cables, tension meters may be required
to ensure cable physical ratings are not exceeded.
Splices on underground systems can be located in above
ground pedestals or below ground. The electric utility
industry typically installs ADSS fiber cable for both aerial and
underground installations. This reduces stores inventory while
mitigating concerns for managing circulating, induced, and
ground fault currents that could be present on armored type
cable that the telecom industry installs.
Regardless of the type of physical properties of the cable –
ADSS, OPGW, Lashed, glass strands are individually color
coded and grouped into a common jacket known as a buffer
tube. Buffer tubes, which themselves are color coded per
EIA/TIA-598, can normally be found with six, 12, or 24
strands. Best industry practices tend to match the number of
fibers in a buffer tube with a corresponding number of
terminating points in a termination module. This practice also
enables the owner of the fiber facilities to lease out dark fibers
‘by the buffer tube’ to third parties to generate extra revenue.
In general most services at a substation can be met by just a
single pair of fibers in a multiplexed or packet-based network.
Other fibers within the cable can be used to support non-
substation corporate applications or leased out to other
companies to generate revenue, or dark fibers (spare fiber not
in use) swapped with other companies to provide redundancy
on the network for one or both companies.
1) Point-to-Point Fiber Connection
Prior to cost effective fiber network equipment becoming
available at the substation, many utilities leased point-to-point
copper-based communication paths for SCADA and protective
9
relaying. The first use of fiber for protective relaying in many
companies involved a direct connection between relays, using
one pair of fibers between two substations.
As depicted in Figure 1.6 below, two IEDs such as line
differential relays can be connected to each other with a single
fiber pair. Each fiber in the pair serves to transport light
energy carrying digital signals at the same wavelength. Since
communication circuits are normally bidirectional, in that each
device needs to both speak and listen concurrently. These
functions are traditionally carried by two separate fibers
between one site to the other at one wavelength, while the
other fiber transports data in the opposite direction, both
typically at the same wavelength.
IED 1 IED 2
TX RX
RX TX
Figure 1.6
Point-To=Point Fiber IED Connection
In some cases where a network is fiber-poor in a certain area,
it is possible to superposition two different wavelengths of
light on one fiber through the use of special prisms at each
IED. This technique is referred to as Wave Division
Multiplexing and is depicted in Figure 1.7 below.
1310nm
1550nm
TX
RX1310nm
1550nm
TX
RXSINGLEFIBER
Figure 1.7
The above technique is more commonly applied at the
network level as opposed to the IED level. Nevertheless,
devices exist which convert and combine a device’s optical
energy into multiple wavelengths.
Several manufacturers offer media converters, enabling
equipment with copper connections to be converted to an
optical signal, with the ability to span the distance between
substations. Many IED manufacturers provide a fiber
interface as an option for communicating short distance. Short
distance reach is normally achieved with less expensive
multimode optics and cable, up to 2km or so. Longer
distances will require the use of singlemode fiber and optics.
At this point in time, distances of 160km are possible using
1550nm class lasers before signal regeneration is needed. The
actual distances depend on the fiber type, optical budget
available in the fiber transmitter-receiver design, and other
factors.
Following is a summary of the benefits and risks of applying a
point-to-point circuit:
Simplicity – fewer devices and work teams
Cost is lower than a networked solution
Control of electronics remains within P&C group
Minimal latency/no asymmetrical delays
No intrinsic redundancy
Inefficient use of fiber bandwidth capability
2) Time Domain Mulitplexing (TDM) Networks
The structure of both TDM networks and packet-based
networks requires ‘Overhead” data that consumes a portion of
the overall bandwidth. The “Payload” portion of the data
frame is the portion which contains the actual data.
In both types of networks, think of the Overhead section as
containing information that facilitates transporting the actual
data to the intended location(s). It may contain information
such as start and stop bits, origination and destination address
information, predefined bit patterns to assist in the recognition
of errors, network switching information, Network
Management information indicating the status and health of
each element in the network, and other functions.
In a TDM network, the payload is fixed and dependent on the
bandwidth that is associated with the equipment. The payload
in a packet-based network can vary between 48 and 1500
bytes.
The origins of the TDM network date to the expansion of the
voice-dominated telephone circuits in the 1950s and 1960s of
communications between central offices to end users. Higher
circuit density and reliability was needed. Original point-to-
point TDM networks, essentially channel banks/multiplexers
running at T-1 rates, formed the core of a new higher density
communications network. Any single hardware failure (cable
and multiplexer equipment) could essentially stop
communications from flowing between sites or regions.
When the Synchronous Optical NETwork (SONET) standard
was created, it enabled ring-protected architectures that have
enabled reliability for all types of circuits, including those
found at substations. Some engineers would argue that
transporting Ethernet circuits across a SONET has advantages
that packet-based networks may still not offer [12].
The ability to further multiplex DS-1 traffic into higher order
networks and bandwidth was defined by the SONET standard,
released in 1984 [13] [14] [15] [16]. The key difference
between the origins of the TDM network and the Packet-based
network is centered on information delivery. TDM networks
have delivered traffic in a deterministic, low-latency method
and also have the characteristic of ‘healing’ after a fiber break
or laser failure relatively quickly.
TDM circuits are based on the basic building block of 64
kbps, DS-0 channels. Inventors in Europe were developing
similar technologies and practices and settled on the same DS-
10
0 building block, however packaged 30 DS-0 channels
together to form the E-1 at a rate of 2.0 Mbps for the
foundational building block of the Synchronous Digital
Hierarchy (SDH) networks. Although the DS-0 building
block is common between these networks, they are not
compatible without data converters at the DS-1/E-1 level.
North America standardized on 24 DS-0 channels named the
DS-1 or T-1 at a total bandwidth of 1.544 Mbps as defined in
the SONET standard. A combination of 28 DS-1’s is the next
higher order of multiplexing for an STS-1 (Synchronous
Transport Signal) rate if the line connection is electrical or
OC-1 (Optical Carrier) if the line connection is optical.
The two tables below provide the foundational hierarchy for
TDM and SONET networks:
Signal Bit Rate Channels
DS-0 64 kbps 1 DS-0
DS-1/T-1 1.544 Mbps 24 DS-0
DS-3 44.736 Mbps 28 DS-1
Table 1.2
TDM Circuit Hierarchy
Table 1.3
SONET/SDH Hierarchy
Whereas the general implication for SONET is that the
network is based on fiber media between sites, microwave
connections can be used up to the OC-3/150 Mbps rate to
connect two nodes, whether point-to-point or as part of a
hybrid network, where there is a mix of fiber and microwave
network segments.
A cautionary statement regarding microwave segments is that
by their nature, microwave networks rely on line-of-sight
connectivity and mechanical and electrical connections
working properly to operate properly. There really is no
practical method to ensure the microwave segment will be
unaffected by low flying aircraft or other obstructions such as
buildings, wind turbines, water towers or other structures
rising along or near the microwave path. Microwave
networks carrying this type of bandwidth are typically
operated in a licensed bandwidth. Obtaining FCC licensing
for the frequency to use is an eight to twelve month process,
and requires annual renewal fees and other inspection
activities. Latency through a microwave network designed to
carry SONET traffic is typically minimal, but an equipment
change for maintenance or upgrade purposes can affect
latency and operational performance.
The SONET framework comes with the capability of
implementing the following network topologies:
Point-to-point
Linear
Ring
Subtended Rings
The point to point network in Figure 1.8 below is essentially
similar to some of the original LEC central office to central
office connections prior to the advent of SONET offering
network restoration. Among the reasons to apply a point-to-
point SONET network is that the application only requires two
sites to be connected. For some marginal reliability gains and
maintenance, connecting the SONET nodes in a collapsed ring
fashion at least guards against laser and fiber patch cable
failures, but provides an alternate path, even though the path
may physically be in the same cable.
MUX 1 MUX 2
Figure 1.8
Point-to-Point Network
A linear network configuration as shown below in Figure 1.9
is quite simply an extension of the point-to-point connection,
but three or more nodes are connected in a line. This type of
connectivity has a relatively low cost, but there is no real
protection or circuit restoration of the network for a failure
along any of the segments. As with the previous example,
using the same physical cable to connect the end-most nodes
together provides some relief for maintenance work or laser or
fiber patch cable failures.
MUX 1 MUX 2 MUX 3
Figure 1.9
Linear Network
A ring topology depicted in Figure 1.10 begins providing the
levels of reliability that SONET was designed to provide.
Low latency, jitter-free, deterministic delivery of traffic
11
between sites is again a fundamental characteristic of TDM
circuits. Fiber path diversity adds a desirable level of
reliability to the network, since for any fiber cable, patch
cable, or laser failure, all traffic is automatically restored.
MUX 1 MUX 2 MUX 3
MUX 1 MUX 2 MUX 3
Figure 1.10
Ring Topology
It is not uncommon in larger utility networks to have multiple
‘subtended’ rings that tie directly into the core network. This
type of design approach is favored above a simple ring
network in that as the network grows larger in both fiber miles
and number of nodes, the risk of failure also increases. The
caution for a protection engineer evaluating subtended ring
design is to properly manage latency for the normal and
variety of network conditions where the ring(s) switch in order
to restore traffic. Latency can increase beyond an acceptable
range in some large networks and for certain network failure
conditions.
A subtended ring may also operate at a lower bandwidth, and
have its bandwidth groomed into the higher order network.
MUX 1 MUX 2 MUX 3
MUX 1 MUX 2 MUX 3
MUX 1 MUX 2 MUX 3
MUX 1 MUX 2 MUX 3
RING #1
RING #2
Figure 1.11
Multiple Interconnected Ring Topology
A substation-hardened SONET network’s inherent capabilities
of providing dedicated bandwidth, low latency, fast switching
for hardware or fiber failures, jitter-free operation, and
symmetrical network switching makes it a preferred means of
delivering relay protection and other circuits. It can also
transport Ethernet circuits with enhanced switching
characteristics of the SONET network.
The SONET standard was written primarily to support the
reliable transportation of carrier-class telephone and data
services. The standard also does not require symmetrical
switching, thus introducing the potential for different Transmit
and Receive times, potentially impacting protective relay
schemes that rely on tight tolerances between Transmit and
Receive delays.
A few key points for the Protection & Control and SCADA
engineers to understand is that the SONET standard requires
network switching to occur in under 50 ms (10 ms to detect a
network issue and 40 ms to switch the network). A typical
carrier-class SONET network element (a single node in the
network) does not have the ability to interface at bandwidths
of less than a DS-1, requiring a channel bank to be added so
that ‘edge’ devices such as protective relays, telephones,
meters, etc., can be multiplexed into the SONET network.
Since this is a synchronous network, the channel bank
equipment must also synchronize end-to-end, which may take
another 60 ms, increasing the potential overall outage time to
110 ms. Protection-class SONET equipment typically has a
range of DS-0 interfaces available that are required at the
substation. This eliminates any outboard resynchronization
delays, and maximizes performance for critical protective
relay circuit. Network switching for relay protection class
SONET networks typically occurs in five milliseconds or less.
Note that no industry standard has been published for such a
network. Relay protection performance criteria have driven
manufacturers to develop protection-class, enhanced
networks.
3) Packet Based Networks
The origins of the Internet and Ethernet communications we
enjoy today date back to 1958 when President Eisenhower
formed the Advance Research Projects Agency (ARPA) in
response to the USSR’s launch of the Sputnik satellite [17].
Just eight years later, in 1966, the ARPA Computer Network
project began (ARPANET) with a goal of creating methods of
moving data between computers. In 1992, the first
commercial Internet access was offered [18]. Since those
early days, the delivery of Ethernet packets has been a ‘best
effort’ approach. The proliferation of Voice-over-Internet-
Protocol (VoIP) with its more demanding circuit performance
requirements has driven packet-based networks to approach
TDM performance levels.
For substation applications, Ethernet for corporate business
access and a high-speed data pipe for Digital Fault Recorders
began proliferating in the late 1990’s as utilities began
leveraging the presence of fiber network access at the
substation. Ethernet access to the substation for virtually any
12
reason is loaded with cyber–security concerns as new
evolutions of NERC-CIP emerge [19].
As with all forms of digital communications, packet based
data is sent in a frame format, with an Overhead section and a
Payload section. The most rudimentary packet based frame is
shown in Figure 1.12 below
Figure 1.12
Basic Ethernet Frame
As Ethernet (or IP) is transported around the network through
a method such as Transmission Control Protocol (hence the
acronym TCP/IP), additional Overhead information is added
as shown below in Figure 1.13:
Figure 1.13
Transmission Control Protocol (TCP) Header
Packet based networks include the family of technologies
including Internet Protocol, X.25, Asynchronous Transfer
Mode, Frame Relay, and MPLS to name a few. These
network types employ a ‘best effort’ delivery of packets, but
may not be completely free of dropped packets, variable
latency (known as “jitter”), or asymmetrical switching. The
interface with a packet based network is not time-based or
synchronized as is a time based network, thus there is no real
ability to guarantee the timely transportation or delivery of a
packet that contains data. The technique of packetizing a T-1
in order to transport serial circuits over a packet-based
network is known as Pseudowire.
The Protection & Control and SCADA engineer should be
aware that using Pseudowire packetization adds latency and
subjects the serial circuits to jitter and potentially
asymmetrical and extended switching times. The presence of
packet based networks in a substation, particularly those that
transport control signals, are coming under increasing NERC-
CIP scrutiny and may require more advanced techniques such
as encryption.
Ultimately, transporting serial circuits over a packet based
network can be full of compromises in circuit performance.
The astute engineer will learn to ask the right questions of the
equipment vendor and the IT-Telecom architect and develop a
comprehensive list of circuit performance requirements to
avoid mis-operations and regulatory agency fines.
Information contained at the substation and at field devices is
seen as more valuable and increasingly necessary. Utilities
are gathering and processing electric system operating
quantities at previously unprecedented levels and becoming
more efficient and responsive in outage restoration and
maintaining the grid. Accessing more information about the
state of the power grid at any point in time has grown to where
conventional serial connections are proving too slow,
cumbersome, or expensive to manage and maintain. Ethernet
networks by design can accommodate near-real-time data
needs quite well and offer acceptable network restoration
times.
On the other hand, power system relaying based trip signals
that are delayed longer than a few handfuls of milliseconds
can have substantial deleterious effects including equipment
damage, lead to grid destabilization, and generating unit trips.
Heavy fines, negative publicity, negative reviews prior to any
rate increases, stock devaluations, and lawsuits are just a few
of the potential implications.
Telecom-class Ethernet equipment is still by and large
incapable of supporting the most stringent of Protection &
Control needs. IT professionals, while increasingly competent
at facing security concerns and taking advantage of
technologies such as MPLS and Carrier Ethernet, function in a
different paradigm of considering ‘what is acceptable’ in
terms of network switching, asymmetrical delays, preferred
path restoration, and even ‘what-if’ analysis that protection
engineers must apply before applying new protection schemes.
The Protection & Control and SCADA engineer should
remain involved in following improvements to packet based
networks, as even the most critical of relay protection circuits
may soon be successfully delivered by the next generation of
protection-class, packet based networks.
II. COMMUNICATION CIRCUIT TYPES
A communication circuit can be as simple as two devices
connected by wire, cable, or radio waves with one device
transmitting data and the other device listening for a
unidirectional communication. A rudimentary transfer trip
circuit can be implemented unidirectionally. Until recently,
communicating “OPEN” or “CLOSE” commands to voltage
or VAR regulating devices in the field were largely made
using one-way commands to the device. Personal
communication pagers were initially one-way devices, with
the ability to only receive data.
IP header TCP header TCP data
Sequence number (32 bits)
DATA
20 bytes 20 bytes
0 15 16 31
Source Port Number Destination Port Number
Acknowledgement number (32 bits)
window sizeheader
length0 Flags
Options (if any)
TCP checksum urgent pointer
20 bytes
13
Most modern forms of serial substation communication exist
in a bidirectional form, with two possibilities of
communicating: Half-Duplex and Full-Duplex. The ability to
speak (“Transmit”) and listen (“Receive”) at the same time is
referred to as full-duplex. The other type of connection,
where only one device may transmit while other devices
receive information is called a half-duplex form of
communication. An example of half-duplex configuration is a
two-wire RS-485 connection for a DNP SCADA circuit,
where a parallel-connected string of Remote Terminal Units
(RTUs) or IEDs on an RS-485 circuit wait for a polling
request from the SCADA Master. Only the IED that
corresponds to the address included in the poll will respond
with the requested information.
Hardware connections described below are dependent on the
capabilities of the protocol and-or driving software in terms of
applying a half-duplex connection vs a full-duplex connection.
From a physical connection point of view, a half-duplex
circuit requires two fewer wire connections to be made,
reducing the cost of design and installation. Further, as
mentioned in the SCADA connection example above, a half-
duplex connection may be appropriate for certain applications
where only one device is required to speak at any one time in
the connected network.
To ensure the most reliable and dependable communications
service as possible, the specific cable required for each
application must be specified and installed. Recommended
cable types will be suggested in each of the following
sections.
A. Analog
1) 4 Wire AC Data
Some of the oldest communications connections to or between
substations are in the form of cables that transport sinusoidal
data. Pilot wire relay schemes transport ac current over wire
pairs. Legacy analog SCADA circuits and audio tone
protective relay circuits have all been transported as AC
circuits.
A reliable and well-performing AC based data circuit requires
methods of matching impedance, frequency-equalizing and
transmit/receive level setting, techniques for reducing induced
power system noise on the line, adequate isolation and
protection against the effects of Ground Potential Rise (GPR)
and becoming a carrier of power system fault currents. It also
requires personnel that are trained and experienced in
designing, installing, and maintaining this technology, which
is rapidly diminishing.
Circuits carrying AC data have characteristically low
bandwidth, and are more difficult to obtain from the local
telephone company. Technology obsolescence and
improvements, the relatively low cost of higher bandwidth
services, the retirement of persons with technical knowledge
of designing and maintaining equipment, and utilities
installing their own networks have all contributed to the
decline of the need for AC Data circuits.
The characteristic impedance of AC Data interfaces is 600
ohms. A cable type that is well suited for this type of
application is a twisted shielded pair cable. Using American
Wire Gauge (AWG) #18 stranded wire for the conductors
offers a cost effective solution that offers a reasonably large
wire size for landing on screw terminal connections, while still
able to be specified as a twisted shielded pair cable. Some
successful applications are possible using pre-terminated,
connectorized cables of smaller wire size.
2) Plain Old Telephone Service (POTS)
“POTS” is the telecommunications industry name for the
standard two-wire analog telephone lines. This circuit can be
transported over a single pair telecommunications cable. It is
the most popular medium used by electric utilities and has
existed as it is still in use today. Compared to other available
options, it has been the most economical way to communicate
with remote locations.
A single POTS line can be used not only for voice
conversations, through a modem, but also for connecting to
Intelligent Electronic Devices (IED) at the substation or in the
field. Bandwidth for dial-up access is limited to 56 kbps, a
single DS-0’s bandwidth on a TDM network.
Many utilities rely on a POTS line from the Local Exchange
Carrier (LEC, a.k.a. “the local phone company”) for E-911
emergency services, as well as data and voice
communications. Equipment needed for data communication-
a dial-up modem- is easy to install, but is quickly becoming
obsolete and removed from service due to cyber security
concerns.
The use of a dial-up phone line for remote data access, even if
the line was provided by a utility’s internal telephone network,
has been subject to NERC-CIP standards and increasing
cyber-security scrutiny. Connections to modems are rapidly
being disconnected from substation IEDs.
Legacy telephone networks have a one-to-one connection
relationship in that for every telephone there is one dedicated
connection to the telephone switch (PBX). This is true
whether the PBX is owned and operated by the utility or by
the LEC (“Local Exchange Carrier” i.e., the “Phone
Company”).
Dial-up telephone service is gradually being replaced by
reliance on cellular technology and “ON-Net” internal phone
connections via utility VoIP servers or PBX, but still serves a
vital role in E-911 service at many utilities. Figure 2.1 depicts
a simplistic architectural view of POTS phone lines and
connectivity with the telephone switch and the E-911 service
provider.
14
Station 1 Phone
Station 2 Phone
Station 3 Phone
Internal SONET Ring Network
CorporateTelephone Switch Patch Panel
Public Switched Telephone Network
l
T-1 Trunk Lines
T-1 Trunk Lines
Patch Panel/Demarc Point
Figure 2.1
POTS Telephone Connection
Note that any copper facilities entering a substation’s “Zone of
Influence” as described in IEEE Standard-487 must be
adequately protected against the deleterious effects of Ground
Potential Rise (GPR) in order to maintain reliability and
personnel safety. Many issues with phone line reliability can
be traced to the lack of adequate protection and isolation as
outlined in the standard.
POTS lines have been successfully connected using Cat3 and
higher cable types, and are typically terminated in an RJ-11
plug.
B. Digital Serial Circuits
In the next sections serial communication circuit types that are
used inside the substation will be discussed.
1) RS-232
RS-232 circuits are widely used for connecting IEDs to IEDs
and IEDs to computers that gather and process data. The full
specification for the RS-232 circuit is defined in
Telecommunication Industry Association (TIA) Standard
TIA-232-F, 1997 and the Electronic Industries Association
(EIA) Standard RS-232-C. The development of these
standards ensures reliable communication between devices
and also interoperability between equipment produced by
different manufacturers. Although two different types of
connectors can be used for RS-232 circuits, the DB-25, and
the DB-9, the most commonly used type is the DB-9
connector.
Information being transferred between data processing
equipment and peripherals is in the form of digital data which
is transferred in either parallel or serial mode. In
telecommunications circuits, RS-232 is widely implemented.
However the circuits are not limited to an RS-232 standard in
that some manufacturers may assign pins for specific
functions, such as powering media converters. In the next
sections some of the different serial communication standards
that are used inside the substation will be reviewed.
Serial transmission involves sending data one bit at a time
over a single communications line in a point-to-point fashion.
In contrast, parallel communications require at least as many
lines as there are bits in a word being transmitted (for an eight-
bit word, a minimum of eight lines are needed). Serial
transmission is beneficial for long distance communications,
whereas parallel is designed for short distances or when very
high transmission rates are required. Officially, RS-232 is
defined as the “interface between data terminal equipment and
data communications equipment using serial binary data
exchange.” This definition defines data terminal equipment
(DTE) as the computer, while data communications equipment
(DCE) is the modem. Some IED manufacturers define all
serial ports as DTE configurations. A modem cable has pin-
to-pin connections, and is designed to connect a DTE device
to a DCE device [20].
Depending on the manufacturer, relays may be equipped with
two or more serial communication ports. One common use of
the serial port is to connect the IED to a computer for local
communication, uploading of settings, downloading of event
files, etc. Rear ports are more commonly used for IED-to-
IED connections. The rear ports are usually connected to one
of many input ports on a communications processor (modern
RTU) which serve as portals for information and control
commands to be passed to the IED processor, and makes it
available to users on an HMI device or through SCADA.
RS-232 requires a “home run” connection, otherwise known
as a point-to-point connection. A home run connection is
made with a cable that is directly tied from one device port to
another, with certain defined pin assignments. For multiple
devices to be connected together, an RS-232 switch must be
used. An RS-232 circuit uses a cable with either a DB-9 or
DB-25 style connector for its communications. A DB-9 Male
connector is shown below in a Data Computer Equipment
configuration.
A manufacturer will specify whether a port is DCE or DTE.
When the configuration for both end devices is known, a cable
can be specified to provide a successful connection. Null
modems in either DB-9 or DB-25 styles are available which
serve to reverse the RX and TX pins as well as the RTS and
CTS pins to assist in connecting DCE and DTE devices
through cables with wiring connected straight-through
between the end connectors.
15
Figure 2.2
DB-8 Connector Pin Out, DCE Shown
A Data Terminal Equipment connection switches the Transmit
and Receive pin assignments as well as the RTS and CTS
pins. DB-25 connections are rarer in substation IEDs, but
have similar functionality. DB-9 to DB-25 adapters are
commonly available.
Figure 2.3
DB-25 Connector Pin Out, DCE Shown
One of the issues with RS-232 communications is that the
signals coming through the TD (pin 3) and RD (pin 2) lines
are both referenced to the ground pin (pin 5). This can cause
problems if the reference point (ground) fluctuates in any way.
This is the reason why the RS-232 trigger voltages are set
relatively high [21], and why in a substation environment it is
advisable to keep cable lengths less than 50 feet and require
fiber isolation for connections to any equipment in the
substation switchyard.
At the RS-232 receiver, Logic 0 has a trigger voltage of +3V
and +12V, Logic 1 has voltage between -3V and -12V.
Switching between negative and positive voltages is referred
to as bipolar data. Note that the hardware handshake lines
operate in the opposite voltage sense to the data lines. For
instance, when a control line is active (logic is equal to 1), the
voltage is in the range of +3 to +12 volts and when it is
deactivated (logic is equal to 0), the voltage is zero or
negative. The range from -3V to +3V is dead zone and used
as a buffer to guard against line noise providing false 0 or 1
data.
RS-232 circuits also have a limitation on cable length. This is
because it does not take advantage of the twisted-pair style
cable which reduces line noise through differential mode
rejection. Using a straight cable, the current flowing through
the cable generates magnetic fields, which causes noise. This
causes the voltage levels in an RS-232 cable to fluctuate,
hence the large range for the different logic states.
A twisted pair cable still causes magnetic fields but the
resulting noise current flows in opposite directions and
therefore cancels itself out. Because of the existence of
straight cables in RS-232 the maximum cable length for a
baud rate of 19200 is 50 feet before the message being sent
starts to corrupt the signal it is transporting. Although a
shielded cable offers a measure of rejection of induced
currents and voltages, induced transients and ground potential
rise between end devices can still impact the circuit
Below is a table referencing the cable lengths based on the
desired baud rate.
BAUD RATE MAXIMUM CABLE
LENGTH
19200 50 feet
9600 500 feet
4800 1000 feet
2400 3000 feet
Table 2.1
RS-232 Maximum Theoretical Cable Length/Data Rate
Although the theoretical limitations of cable length are
indicated above, the practical limitation of RS-232 copper
cable in a substation environment is limited to 50 feet
regardless of data rate. Copper data cables can be installed in
or near the same cable tray system as the power and control
cables, thereby increasing the risk of inducing transients on
the data cable. This can lead to data errors or microcircuit
failure internal to the serial port. Damage related errors may
16
not show up immediately after a transient event, making
troubleshooting more difficult. In addition, the “COM” pin is
a return signal conductor and is referred to ground. This can
also greatly affect signal levels from one side of a control
building to another during transient events involving station
ground.
In any data transmission the devices that are communicating
with each other must have a way to extract individual
characters or blocks (frames) of information. When the sender
(typically DCE) and receiver (typically DTE) devices
exchange data the characters arrive in a continuous or serial
stream of bits so you need a way to separate one block of bits
from another. In asynchronous communications as is typical
of a digital serial circuit, each character is separated by the
equivalent of a flag so the receiver knows exactly where
characters are located. In synchronous communications, such
as is a TDM network starting at a DS-1 level, both the sender
and receiver are synchronized with a clock, or a clock signal
encoded into the data stream. There is no shared clock for
sync purposes between the sending and receiving device when
using RS-232 (asynchronous communication). Successful
RS-232 communication circuits need to have data rates of the
sending and receiving devices to be the same or auto-detecting
in order to function properly.
Another requirement for a successful RS-232 communication
is the occurrence of handshaking, whether it is hardware or
software in nature. Hardware handshaking takes place
between the RTS (request to send) and the CTS (clear to send)
pins. A device ready to send indicates it has a message by
pulling the RTS signal line in the positive voltage range (logic
0 or space). The receiving device acknowledges this, and
gives permission for the sending device to send by pulling its
CTS line in the positive voltage range (logic 0 or space). If at
any time the receiving device is interrupted in its receipt of the
message, it will pull the CTS line to return to logic 0 to
continue sending its message.
Software handshaking is different from hardware handshaking
in the fact that two special ASCII characters (XON and
XOFF) are sent in the data line to indicate a message being
sent. When data is ready to be sent, the receiving device sends
the XON character to the sending device to let it know that it
is ready to receive the message.
RS-232 uses a particular frame for sending its messages called
the 8N1 frame. The 8N1 frame is used to make sure that the
receiving device correctly receives the message, as it requires
a space (logic 0) at the end of the frame called a Stop Bit;
otherwise a framing error will occur and both devices will
know that the “sent message” was not successfully received. A
typical 7N1 frame for the RS-232 is shown below in Figure
2.4:
Figure 2.4
7N1 Serial Data Frame
Note that it is possible to deliver RS-232 messages with frame
configurations other than 7N1. The engineer must consult
with the IED manufacturer’s port and software specifications,
and ensure that the network port interface is set to accept the
data frame configuration and data rate in order to properly
configure the port to achieve successful communications.
When transporting an RS-232 circuit over a communications
network, it is critical that the ports on any channel banks or
Pseudowire equipment have the proper hardware and data rate
configuration settings in place in order to transport the signals
properly. The engineer must also be aware if a device uses
hardware or software handshaking to manage the data flow
and enter those settings appropriately.
2) RS-485/422
These two names are the common names for two serial
communications standards. These standards are defined by the
Electronics Industry Association (EIA) and the
Telecommunications Industry Association (TIA) and are more
correctly named EIA/TIA-422 and EIA/TIA-485. An RS-
485/RS-422 circuit can be defined as the configuration behind
the communication interface being used by the receivers and
transmitting devices on which these are implemented. The 422
and 485 standards are balanced data-transmission schemes
that offer solutions for transmitting data over long distances
and noisy environments
However these standards don’t specify a logical
communication protocol. Systems connected using RS-
422/485 interfaces can communicate at rates up to 10 Mbps
(though most systems operate at lower bit rates). Both circuit
types utilize balanced outputs and differential inputs, which
provide better noise immunity than RS-232 circuits. This
results in the ability to operate over longer distances at higher
bit rates than links using RS-232. RS-232 and RS-422 support
full-duplex communications, while RS-485 supports half-
duplex communication. Full-duplex communications allows communication in both
directions simultaneously whereas half-duplex communication
allows communication in one direction at a time. Full-duplex
communications has more advantages than half-duplex
communication, but because of differences in the electrical
properties, the communication distance of RS-232 is less than
15m, and the maximum communication of RS-485 is up to
1200m [22]. Practical substation applications would dictate
the use of fiber connections for any circuits extending beyond
the control house into the substation switchyard or to another
control building due to ground potential differences.
17
An RS-422 connection requires at least two twisted pairs of
wires and it is limited to a multi-drop application. In multi-
drop applications only one transmitter (driver) is connected,
with the ability to transmit on a bus with up to 10 receivers.
RS-485 circuits have the advantage of anti-jamming and far
communication distance characteristics. Thus, they are used
widely in industrial control equipment circuits. RS-485
circuits can be applied in point-to-multipoint applications such
as SCADA, making it a unique cost-saving for connectivity in
a substation where multiple nodes connect with each other.
RS-485 is not limited to a multi-drop connection as it allows
up to 32 devices to communicate through the same data bus
allowing receivers communicate with each other without
having to go through the master. In simpler words RS-485 is
the only of the interfaces capable of internetworking multiple
transmitters and receivers in the same network.
RS-485 circuits have the advantage of rejecting external
common-mode noise and enabling longer communication
distance connections. Thus, they are used widely in industrial
control equipment circuits. In the substation environment,
RS-485 circuits can be applied in point-to-multipoint
applications such as SCADA, making it a unique cost-saving
for connectivity in a substation.
RS-485 and RS-422 circuits are typically connected using
twisted shielded pair cable. Terminations are most commonly
made to either screw terminal connections or DB-9
connectors. DB-9 connections are more common where the
port can be configured for different types of interfaces such as
RS-232 or RS-485.
a) IEC-61850
An IEC-61850 network is Ethernet based. Many utilities have
studied and considered applying this method of connecting
devices together in a local network. Some utilities in the USA
are implementing the technology, but broad acceptance
appears years away. 61850 networks appear well suited for
highly standardized schemes or systems with little variability
between sites. For this reason, substations at large refineries
and other similar installations with perhaps several dozen or
more substations being essentially of the same vintage and
having the same operational design requirements, may benefit
most from the application of 61850 technology.
Network devices such as Ethernet switches and routers must
be 61850 compliant in order to be used on the network. Two
of the key performance related parameters such as Quality of
Service (QoS) and Virtual Local Area Network (VLAN), need
to be available as settings on the Ethernet switches, routers,
and IEDs to the designer of a 61850 network in order to
optimize data flow on a 61850 network. Network equipment
that is not specifically IEC-61850 compliant will not be able
to transport packets containing 61850.
Figure 2.5
Simple IEC-61850 Example Topology
Temp Source: kapadia_gi10.pdf
While implementing the technology can present technical
challenges, it can also present serious organizational
challenges. Programming of all routers and switches may fall
under the direction of an organizations’ IT team, who may not
have electric grid operational expertise. Additionally, IT
experience has traditionally not been a core competence of the
Protection & Control engineer, which places some additional
training opportunities for those involved in the planning,
design, implementation, and maintenance of an IEC-61850
system.
A 61850 network effectively marries the two disciplines in
previously unnatural ways in that 61850 requires SONET-like
performance which is typically difficult to achieve in telecom-
class network equipment. Although the transition to a 61850
network can be successful, the decision cannot be casual; it
must be supported at the directional levels of an organization
with a comprehensive business case attached, and the
appropriate training and close coordination between affected
departments. The technical aspects of 61850, it’s features and
functions, and case studies have been well represented in other
documents.
Some utilities mandate that any device with network
connectivity be under the direction of an IT team. Research
on the application of 61850 on a global level, shows how
dominantly it is applied in the rest of the world. Discussions
of the IEC-61850 standard, indicate it is not yet complete, and
somewhat subject to individual manufacturer interpretation.
Since IEC-61850 connections are Packet-based, the cable type
used would be Cat5e or Cat6 cable terminated to RJ-45 plugs.
Four pairs of individually twisted pairs of conductors are
present in a Cat5e or Cat6 cable. Although twisted pair rejects
common mode noise fairly well, some utilities choose to
specify shielded Cat5e or Cat 6 cables as a standard.
b) VoIP
The emergence of Packet-based networks across many utility
substations has led to the deployment of Voice-over-Internet-
18
Protocol (VoIP) not only to office locations, but also to
substation sites. Adding a new VoIP phone extension to a site
that is on the utility’s corporate network can be as simple as
providing an Ethernet port to the site and tapping into the
bandwidth that’s tied in to the company’s VoIP server, which
is typically installed at a data center location. While some
programming is required at the VoIP server to enable the new
service, no new physical connections are required except at
the site where the new “Telephone Appliance” is located.
This represents a tremendous cost savings for any new
connections, both in labor and materials.
VoIP Server
Station 1 VoIP Phone
Firewall-Router Station 2 VoIP Phone
Station 3 VoIP Phone
Internal IP over SONETOr
Packet-Based Network
Public Switched Telephone Network
Commercial Core IP
Network
Figure 2.6
Simple VoIP Topology
Although the use of modems in the substation has
dramatically reduced over the past few years due to the
increased security risks, the transition to VoIP is essentially
transparent to any remaining dial-up modem applications in
use at the station.
A compressed VoIP connection requires approximately the
same bandwidth as a telephone circuit, about 64 kbps (a DS-0
in a Time Division Multiplexed network). The actual
bandwidth depends on the type of compression applied, the
type CODEC, and other factors which a VoIP architect or
manufacturer would be designing. For a more thorough
discussion please refer to [22].
One concern with VoIP or POTS lines served by either a
utility’s internal telephone server or PBX relates to the ability
to “Dial 911” and have the address of the actual site of the
problem appear on the dispatcher’s screen.
For phone calls destined for a phone extension on a Corporate
internal phone network, it is not uncommon for three or four
digit dialing to be enabled. Upon going off-hook, the dial tone
originates from the Corporate PBX or VoIP server and
provides toll-free access to internal extensions. After going
off-hook and dialing “9”, the “9” directs the PBX or VoIP
server to obtain a connection with the Publix Switched
Telephone Network, which enables “metered” calls across the
world through the Public Switched Telephone Network
(PSTN).
From a substation safety point of view, the company’s PBX or
VoIP server might be hundreds of miles away from the actual
telephone and where assistance is needed. When an external
dial tone is reached, the physical location of the PBX or VoIP
server is typically passed along to the Emergency Response
dispatch office unless special steps are taken. This can add
long delays in the process of dispatching first response units to
the scene of the need.
A company’s Telecom team coupled with the appropriate
substation business unit needs to carefully consider E-911
response procedures; even considering cellular phone initiated
911 calls. Many utilities are proactively installing station
location details inside the control building and at the
substation, and making workers aware by procedure of the
site-specific emergency procedures.
c) IEEE C37.118 Synchrophasors
Synchrophasor technology has significantly developed over
the past ten years, when it was available primarily on
dedicated IEDs known as Phasor Measurement Units (PMUs).
Interest in applying Synchrophasors escalated rapidly after the
blackout in the northeast USA in August 2003. Today, many
protective relays include the PMU function to measure and
compare voltage magnitude reference to an absolute time.
Additionally, time stamping allows Synchrophasors from
different utilities to be time-aligned (or “synchronized”). A
Phasor Data Concentrator (PDC) collects Phasor data from
connected components in the system and prepares the
information in a time-aligned manner for applications to
process. This provides a precise and comprehensive view of
an entire interconnection. Synchrophasor technology enables
a better indication of grid stress, and can be used to trigger
corrective actions to maintain reliability as well as a variety of
merging applications [23].
Phasors are defined as a magnitude of quantity about its
rotation, specifically, an angle with respect to a reference
vectors. Synchrophasors are regarded as an angle in a specific
point in time. A synchronous phasor set of electric quantities
in the transmission network is denoted as a phasor snap of the
system [24]. One significant difference between SCADA
analog observations and Synchrophasor technology is that
SCADA performs a system observation once every four
seconds depending on a utility’s practices. Phasor data can
have measurement rates between 60 and 240 times per second,
providing a much better detailed image of the grid.
Synchrophasors can be implemented to increase reliability of
the power system as a whole by using its features with Wide
Area Monitoring (WAM), with Smart Grid applications, and
real time operations to name a few.
19
Wide Area Monitoring (WAM) is a complex system that
encompasses PMUs connected to a Phasor Data concentrator
(PDC), typically centrally located at an energy control center
or it’s associated data center. All Phasor data is collected and
transmitted to the PDC providing operators with real time grid
conditions, anticipate changing conditions, and quickly
implement actions to protect system’s reliability. Phasor data
can be transmitted to the PDC over serial or Ethernet
networks. Ethernet’s characteristic of transporting many
applications in a multipoint-to-multipoint manner makes it an
often preferred method for delivering Phasor data. The
mission critical task is for the voltage and angle relationship to
be time stamped with high accuracy as defined in IEEE
C37.118 which is the most dominant protocol for transmitting
Synchrophasor data.
Synchrophasor implementations commonly use IP networks to
deliver Phasor data with User Datagram Protocol/Internet
Protocol (UDP/IP), or Transmission Control Protocol/Internet
Protocol (TCP/IP) when a large number of PMUs is needed.
Phasor data can also be transported using RS-232 serial
connections for networks that have a small number of PMUs
reporting to a correspondingly high number of individual
connections at the PDC, potentially raising the cost of
implementation and maintenance.
One critical requirement for a successful Synchrophasor
implementation is to apply accurate time stamps at each PMU
in order forthe applications running at the PDC to properly
time align Phasor data from locations around the grid. An
emerging technique is to use a network connected, highly
accurate terrestrial time source, to ensure that a single GPS
antenna failure or a regional GPS outage does not lead to loss
of Phasor data.
Phasor data can arrive at the PDC within typical TDM or
packet based network latencies in which the data is
transported. The deciding factor for how much network
latency can be tolerated depends on how the Phasor data is
used. Synchrophasor records can be studied for the purpose of
enhancing reliability by studying trends or maintaining grid
stability when used in a real-time manner. Real-time or near-
real-time applications will prescribe a network with lower
latency than Phasor data used for historical trend analysis.
Real time applications include the following benefits:
Frequency stability & power oscillation monitoring
Voltage monitoring
Event detection
Outage restoration
Alarming & operating limits
State of the grid
Planning applications include:
• Trend analysis
• Event analysis
• Calibration and validation models for Static and
Dynamic systems
• Power system performance
• Power plant modeling
• Load attributes
• Special protection schemes (SPS)
In the US alone, over 15,000 relays and Digital Fault
Recorders are deployed. Many can be upgraded to include the
PMU function with a firmware upgrade.
The long-term challenge for Synchrophasor technology is to
prove its value for operations and planning, to validate
industry investment and ownership in production-grade, fully
utilized systems [24] [25]. NERC regulations are driving
utilities throughout the US not only to adopt Synchrophasor
technology but also to develop applications and validation
tools to increase grid stability and reliability, and perhaps
shorten restoration times after system events [26].
d) Precision Time Protocol, IEEE-1588
As is the case with enabling features in a Packet-based
network, the structure of each individual packet of information
must be capable of transporting not only the data, but carrying
and acting upon the options that a particular feature offers.
Precision Time Protocol as defined in IEEE Standard 1588 is a
method by which sub-millisecond accuracy time stamps can
be transported across a Packet-based network. Within the
protocol itself rests a dynamic process of evaluating the delays
between the source Grandmaster clocks and subsequent
Ethernet network elements and applying algorithms which
compensate for latency between devices along specific
segments. The time stamp is embedded within the Ethernet
header.
Figure 2.7
Calculating delays between clocks [27]
20
PTP time is established at one or more server locations around
a network. Those servers have the function of Grandmaster
clocks within the network. The ideal network would have
multiple connections between routers on the network, which
are able to time-correct the time stamps and present a time
stamp at devices connected to a router or switch which has
been compensated for measured delays.
PTP is not in widespread use across a utility’s substation
network at this writing. This technology is more commonly
found in process-oriented manufacturing environments. The
PTP standard was primarily developed to support the
proliferation of Phasor Measurement Units and Ethernet
networks in the electric utility environment.
e) DNPoIP
Distributed Network Protocol version 3 (DNP3) is a protocol
for transmission of data from point A to point B using serial
communications. Used mainly by electric utilities, DNP3 is
specifically developed for inter-device communication
involving SCADA Remote Terminal Units (RTU), and
provides for both RTU-to-IED and Master-to-IED. It is based
on the three-layer enhanced performance architecture (EPA)
model contained in the IEC 60870-5 standards, with some
alterations to meet additional requirements of a variety of
users in the electric utility industry. DNP3 was developed with
the following goals: High Data Integrity, Flexible Structure,
Multiple Applications, Minimized Overhead, and Open
Standard. It provides the rules for substation computers and
masters station computers to communicate data and control
commands. Figure 2.8 shows the master-outstation
relationship and gives a simplistic view of the databases and
software processes involved (Master is on the left side,
outstation is on the right side) [29] [30].
Figure 2.8
DNP3 Client-Server Relationship [31]
A series of square blocks at the top of the outstation depict
data stored in its database and output devices. The various
data types are conceptually organized as arrays. An array of
binary input values represents states of physical or logical
Boolean devices. Values in the analog input array, represent
input quantities that the outstation measured or computed. An
array of counters represents count values, such as kilowatt
hours, that are ever increasing (until they reach a maximum
and then roll over to zero and start counting again.) Control
outputs are organized into an array representing physical or
logical on-off, raise-lower and trip-close points. Lastly, the
array of analog outputs represents physical or logical analog
quantities such as those used for setpoints. The elements of the
arrays are labeled 0 through N - 1 where N is the number of
blocks shown for the respective data type. In DNP3
terminology, the element numbers are called the point indexes.
Indexes are zero-based in DNP3, that is, the lowest element is
always identified as zero.
The master and the outstation shown in Figure 2.8 each have
two software layers. The top layer is the DNP3 user layer. In
the master, it is the software that interacts with the database
and initiates the requests for the outstation’s data. In the
outstation, it is the software that fetches the requested data
from the outstation’s database for responding to master
requests. It is interesting to note, that if no physical separation
of the master and outstation existed, eliminating the DNP3
might be possible by connecting these two upper layers
together. However, since physical or possibly logical
21
separation of the master and outstation exists, DNP3 software
is placed at a lower level. The DNP3 user’s code uses the
DNP3 software for transmission of requests or responses to
the matching DNP3 user’s code at the other end. Figure 2.9
shows the DNP3 architectural layers.
Figure 2.9
DNP3 Architectural Layers Source [31]
In recent years, IED manufacturers have begun offering
Transport Control Protocol/Internet Protocol (TCP/IP) to
transport DNP3 messages in addition to the legacy serial DNP
connections. Link layer frames are embedded into TCP/IP
packets. This approach has enabled DNP3 to take advantage
of Internet Technology and permitted economic data
collection and control between widely separated devices. To
be able to do this approach manufacturers have been including
Ethernet ports on IED’s. Although an Ethernet port may be
rated for 10/100 Mbps bandwidth, the data throughput is
dependent largely on the processing capability of the
communications processor in the IED and may be orders of
magnitude less than 10 Mpbs.
TCP/IP provides a level of error detection in that it has certain
bits in the overhead that have a certain pattern that is expected
to be received accurately. If the pattern is not received, a
rebroadcast request is typically sent. The application layer,
DNP in this case, also has some built in error detection. For
SCADA purposes, if a packet is corrupt the SCADA system
would normally just wait for the next scan, since SCADA
protocols rely on a round-robin approach to polling RTUs at
substation sites. After being polled for information or sent
control commands, the RTU only has a short window of time
in which to respond. If the SCADA Master does not receive
good response (could be from communications circuit failure
or packet corruption), the SCADA system will get around to
polling that same RTU again, usually within a few seconds.
Subsequent failures to receive good information will generate
an alarm and lead a technician to be dispatched to
troubleshoot.
From the above example, although TCP/IP can generate
rebroadcast requests, the SCADA protocol continues its round
robin polling. If the speed of communication and network
bandwidth are fast enough, there may be enough time built in
to a polling cycle at an RTU that the data packet could be sent
out twice, but it may not occur as often as a good response is
achieved in the next round of polling. There may be anywhere
from 2 – 20 RTUs on any one communications port on a
SCADA Master system, and multiple hundreds of RTUs on
multiple SCADA Master communication ports all working in
parallel.
As a quick comparison, Figure 2.10 shows the DNP3 protocol
stacks for Serial and IP.
Figure 2.10
DNP3 Protocol Stacks [32]
22
C. Copper vs Fiber Connections
1) Comparisons
Copper Fiber
High Cost (cost of metals
has been increasing)
Moderate Cost
(continuously
decreasing)
Easy connections Splicing fiber requires
special training
No special tools needed
to make connections
Special tools needed to
splice
Inflexible network
configuration
Inflexible network
configuration
No licensing requirement No licensing requirement
Subject to breakage and
water ingress
Subject to breakage and
water ingress (freezing
conditions)
Subject to
Electromagnetic
interference
Immunity to
Electromagnetic
interference
Relatively high channel
capacity for short
distances
High channel capacity
Installed in utility owned
land or structures but
right-of-way clearance
required for buried cable
Installed in utility owned
land or structures but
right-of-way clearance
required for buried cable
Low cost test equipment Expensive test
equipment
Subject to ground
potential rise due to
power faults and
lightning
Resources to design and
maintain (persons and
materials) higher cost,
and diminishing in
availability.
Immunity to ground
potential rise
Resources widely
available to design,
install, and maintain at
lower cost.
Table 2.2
Copper-Fiber Connectivity Comparison
The cost of metals has been on the rise while prices for Silicon
(a core component of fiber optic strands) and other related
materials has decreased. Since fiber connections offer so
many other benefits such as RFI and transient immunity,
greatly increased bandwidth and trending toward lower cost, a
design engineer now has a more competitive choice between
copper and fiber connectivity for short distances where copper
may have been the legacy preferred method of connection.
III. COMMUNICATING WITH INFORMATION TECHNOLOGY
AND TELECOM ENGINEERS
A. Strategically Planning Substation Networks
It is inevitable that if a substation’s communication
applications are not yet transported or directly served with a
packet based/Ethernet service, the next iteration of a
substation network will likely be packet-based.
An Information Technology architect develops Packet-based
networks around a different set of criteria than is normally
considered for a Protection-Class network. Packet-based
networks were originally developed to support the transfer of
data between computers. Protection & Control engineers
would argue that a modern substation is designed with mostly
computers performing all operational aspects of substation
functions. Whether an IED is a differential relay, a current
differential relay, a meter, or some form of substation
automation or SCADA device, these devices are all
microprocessor driven. Today, all of these devices are
available with Ethernet ports. Through an implementation of
IEC-61850 protocol, even the Ethernet ports can be used for
critical protection applications.
NERC-CIP is certainly exerting another set of criteria on the
IT network architect, and certainly impacts the ability of a
utility engineer or technician to access and manage IEDs,
particularly on the Bulk Electric System (BES). Certain
network designs or security features may be implemented with
potentially deleterious impacts on time-sensitive circuits.
Network upgrade or maintenance work can render certain
applications such as current differential protection, virtually
unusable.
It is mission critical that the Protection & Control engineer
become familiar with the IT and Telecom engineers and
architects within the organization. Document each type of
application that is used in a substation. Document the critical
performance criteria, including jitter tolerance, tolerance for
asymmetrical delays, network switching times, network
restoration times. This information will enable the IT-
Telecom Architect to make informed decisions to facilitate the
design of a communications network that meets the most
demanding communication needs in a substation.
IT and Telecom practices related to network maintenance can
be quite different from those a Protection & Control or
SCADA engineer are familiar with, especially in terms of
willingness to accept contingency risks. For example, it
would be highly undesirable for a Telecom contractor to
replace a set of 48V DC batteries during peak load conditions
or during times of electric system contingencies. The IT-
Telecom professional may be completely disconnected from
daily or even planned grid conditions, contingencies, and
operations.
At a minimum, the Protection or SCADA engineer should
document the following attributes for each type of circuit:
23
Maximum Latency
Maximum Asymmetrical Delay
Maximum Tolerated Network Switching Circuit
Outage Time
Maximum Tolerated Network Switch-Back Circuit
Outage Time
In addition, representatives from the Transmission and
Distribution electric system operations team should be present
at IT-Telecom “Change Management” meetings, where
planned maintenance or construction activities on the IT and
Telecom networks are discussed.
With closely coordinated and well communicated efforts, a
comprehensive, practical, and cost effective network can be
architected, designed, implemented, and maintained which is
NERC-CIP friendly, is future-friendly for emerging
technologies such as IEC-61850 or Synchrophasor
applications, is substation hardened, and can transport and
switch relay protection circuits with the appropriate
performance levels.
IV. REFERENCES
[1] Tampa Bay Interactive Inc. 2004. [Online]. Available:
http://telecom.tbi.net/ddsops.htm. [Accessed 4/2/2013]
[2] David Willis. 1996. [Online]. Available:
http://www.networkcomputing.com/netdesign/frame1.htm
l. [Accessed 4/4/2013]
[3] D. Davis, "Wentworth Institute of Technology," 2004.
[Online]. Available:
http://myweb.wit.edu/wuj1/comp690/MPLS_vs_IPSec.d
oc. [Accessed 26 March 2013].
[4] J. Rushton, "Proceedings of the IEE - Part A: Power
Engineering," in Volume 108 Issue 41, 1961.
[5] "Radio Electronics," [Online]. Available:
http://www.radio-electronics.com/info/wireless/wi-
fi/80211-channels-number-frequencies-bandwidth.php.
[Accessed 26 March 2013].
[6] "Radio Electronics," [Online]. Available:
http://www.radio-
electronics.com/info/wireless/wimax/wimax.php.
[Accessed 26 March 2013].
[7]
[8]
[9]
Silverspring Networks, Why Unlicensed Spectrum,
National Communication Spectrum, 2004
K. Fehrenbacher, Writer, Smart Grid Debate: Licensed
vs. Unlicensed Wireless Spectrum. [Performance]. 2009
VSATS: Very Small Aperture Terminal Systems by John
L. Everett.
[10]
Satellite Usage in the Utility Industry: Experience from
the Field and UTC Study Results. [Online] Available:
www.UTC.org & www.idirect.net [Accessed 4/4/2013].
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
R. M. a. M. Goraj, "Ethernet for IEC 61850," PAC
World Publication, 2010.
Time Division Multiplexing [Online]. Available:
http://en.wikipedia.org/wiki/Time-
division_multiplexing#History. [Accessed 4/4/2013]
Telcordia GR-253-CORE, SONET Transport Systems:
Common Generic Criteria. Issue Date October 2009.
Telcordia GR-499-CORE, Transport Systems Generic
Requirements (TSGR). Issue Date November 2009.
ANSI T1.105.07 1996 (Reaffirmed 2005). SONET –
Basic Description including Multiplex Structure, Rates,
and Formats.
ANSI T1.119.02 1998 (Reaffirmed 2004). SONET –
Operations, Administration, Maintenance, and
Provisioning (OAM&P) – Communications.
THINK Protocols, "A Technical History of the
ARPANET," 26 February 2002. [Online]. Available:
http://www.cs.utexas.edu/~chris/think/ARPANET/Resou
rces/. [Accessed 26 March 2013].
Brad Stewart. Historical Look at Internet Speed
[Online]. Available:
http://www.tylotimes.com/2011/11/historicallookatintern
et/. [Accessed 4-8-2013]
North American Electric Reliability Corporation,
Reliability Standards. [Online]. Available:
http://www.nerc.com/page.php?cid=2|20 [Accessed 4-8-
2013]
Omega, "The RS-232 Standard," [Online]. Available:
http://www.omega.com/techref/pdf/RS-232.pdf.
[Accessed 26 March 2013].
"D-sub 9 connector Pinout and RS-232 Maxmum Cable
Length," [Online]. Available: http://www.db9-
pinout.com/. [Accessed 26 March 2013].
Computer Weekly, "VoIP Bandwidth Fundamentals,"
June 2007. [Online]. Available:
http://www.computerweekly.com/feature/VoIP-
bandwidth-fundamentals. [Accessed 26 March 2013].
P. J. Zawada, "A Survey of Substation Communications
Technology," IEEE, Columbus, 2000
24
[24]
Demeter, E.; Faried, S.O; Sidhu, T.S; Signal phase
shifting during Synchrophasor measurements. Electrical
and Computer Engineering, 2005. Canadian Congerence
on, 1-4 May 2005.
[25]
Schweitzer, E.O.; Whitehead, D.E., "Real-world
synchrophasor solutions," Protective Relay Engineers,
2009 62nd Annual Conference for , vol., no., pp.536,547,
March 30 2009-April 2 2009
[26]
[27]
[28]
[29]
[30]
[31]
[32]
Kasembe, A.G.; Muller, Z.; Svec, J.; Tlusty, J.; Valouch,
V., "Synchronous phasors monitoring system application
possibilities," Electrical & Electronics Engineers in
Israel (IEEEI), 2012 IEEE 27th Convention of , vol., no.,
pp.1,3, 14-17 Nov. 2012
PSRC H11 paper “Exploring the IEEE Standard
C37.118-2005 Synchrophasors for Power Systems”
End Run Technologies. PTP/IEEE-1588 Frequently
Asked Questions. [Online]. Available:
http://www.endruntechnologies.com/ptp-ieee-1588-
faq.htm. [Accessed 4-8-2013]
D. J. Marihart, "Communications Technology Guidelines
for EMS/SCADA Systems," IEEE Transactions on
Power Delivery, vol. 16, no. 2, pp. 181-188, 2001
DNP, "DNP3 Protocol Primer," 20 March 2005.
[Online]. Available:
http://www.dnp.org/AboutUs/DNP3%20Primer%20Rev
%20A.pdf. [Accessed 26 March 2013].
National Communications System, "Technical
Information Bulletin 04-1," October 2004. [Online].
Available:
http://www.ncs.gov/library/tech_bulletins/2004/tib_04-
1.pdf. [Accessed 26 March 2013].
IEEE Std 1615 – 2007. IEEE Recommended Practice for
Network Communication in Electric Power Substation
Other interesting and informative related documents:
S. Ward, "Inside the Cloud - Network Communications
Basics for the Relay Engineer," IEEE, 2008.
Telephony World, "Data Communications Milestones,"
25 October 2004. [Online]. Available:
http://telecom.tbi.net/history1.html. [Accessed 26 March
2013].
Telephony World, "An Overview of the DDS System,"
Tampa Bay Interactive, Inc, 25 October 2004. [Online].
Available: http://telecom.tbi.net/ddsops.htm. [Accessed
26 March 2013].
Federal Communications Commission, "FCC," 22
October 2008. [Online]. Available:
http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-
08-244A1.pdf. [Accessed 26 March 2013].