root cause analysis for 428 massachusetts avenue network event on march 16, 2015 rca for...

Root Cause Analysis for

428 Massachusetts Avenue Network Event on

March 16, 2015

Prepared by James Sadtler Director, Transmission Field Operations

Issued on May 6, 2015

IPL Witness JAS Attachment S1 IURC Cause Nos. 44576 - 44602

RCA for 428 Massachusetts Avenue Network Event

Table of Contents

Contents

1.0 Executive Summary ...................................................................................................... 3

2.0 Introduction ................................................................................................................... 4

3.0 Background ................................................................................................................... 7

4.0 Sequence of Events ....................................................................................................... 9

5.0 Investigation and Analysis .......................................................................................... 10

5.1 Downtown Network Supervisory Control and Data Acquisition (SCADA) .......... 13

5.2 Primary Termination Chamber Observations Post Event ....................................... 17

6.0 Conclusions ................................................................................................................. 18

7.0 Next Steps ................................................................................................................... 20

Footnotes…………………………………………………………………………………21



1.0 Executive Summary

On Monday, March 16, 2015, an electrical flashover occurred in the primary termination chamber of Indianapolis Power & Light Company’s (“IPL”) Edison Underground (“UG”) Feeder 442 at 428 Massachusetts Avenue. This flashover created an electrical fault that lasted approximately 0.075 seconds. The system protection equipment worked as designed and since the primary termination chamber was filled with FR3 fluid, which has lower volatility and higher flash point, the resulting flame associated with the fault was not sustained. All 23 of the closed network protectors connected to the Edison UG 442 circuit operated as designed by opening. This prevented back feed from the low voltage side of the network transformers to the primary termination chamber fault location. In summary, the impact of the incident was minor because the equipment worked as designed. Also, no customers lost power as result of the incident.



2.0 Introduction On Monday, March 16, 2015 at 8:42 PM EDT Edison Underground (“UG”) Feeder 442 relayed. Indianapolis Fire Department (“IFD”) logged an incident/alarm at 8:45 PM EDT for 430 Massachusetts Avenue and arrived at the location 8:46 PM EDT. IPL received a call from IFD that was logged at 8:48 PM EDT of the reported incident and IFD requested assistance from IPL. This location is in the IPL Edison East secondary network. Observers reported a loud bang and a flash but not a sustained oil fire. The event involved an electrical flashover in the primary termination chamber on the UG 442 Network transformer in the two bay underground vault located at 428 Massachusetts Avenue. The nameplate rating of the UG 442 transformer is 500 kVA, 13,200 V (delta) to 216 V (wye) /125 V. The flashover was from the “A” phase conductor to ground. The flashover created an electrical fault which lasted approximately 4.5 cycles (0.075 seconds) with a current magnitude of approximately 5,700 amps. Upon arrival IFD observed light smoke coming from an underground vault. IFD noted a small fire (non-oil fire) in the vault. The fire burned itself out and the event resulted in no customer outages. The primary termination chamber on the UG 442 transformer at 428 Massachusetts Avenue was retro-filled with FR3 dielectric fluid. The retro-fill of the termination chamber with FR3 fluid was a commitment made by IPL in its January 2012 Response to: Independent Assessment of Indianapolis Power & Light’s Downtown Underground Networki. FR3 fluid has lower volatility and higher flash point than the mineral oil that was previously used as the dielectric fluid. The FR3 fluid performed as designed by not sustaining combustion. IPL personnel were nearby when the incident occurred and responded to the scene within minutes of the first report, and were able to determine that since the fire had burned itself out and there were no customer outages, it was not necessary to activate IPL’s full “Downtown Underground Network Emergency Response Plan”. However, the initial steps of the plan, as noted below, were used by the onsite IPL personnel.

1. Established the following roles and responsibilities: • Incident Commander • Public Information Officer • Liaison Officer • Field Operations

2. Preservation of evidence 3. Prepared and sent a message to the IURC 4. Prepared and sent a message to the “Indy News Desk”



Exhibit 1 below shows the layout of the IPL Secondary Network Facilities in the area of the event and lists the two primary underground circuits (UG 412 & 442) supplying the two bay vault located at 428 Massachusetts Avenue. Exhibit 1 – IPL Secondary Facilities in the Area of the Event



Exhibit 2 is an aerial view of the two bay vault at 428 Massachusetts Avenue. The northeast bay contains the Network transformer connected to the Edison Substation Underground 442 Circuit. The southwest bay contains the Network transformer connected to the Edison UG 412 Circuit. Both Network transformers are 500 KVA, 13,200 V(delta) to 216 V(wye)/125 V and supply the Edison East Secondary Network. Exhibit 2 – Aerial View of the Area of the Event



3.0 Background As part of the two year inspection cycle for vault equipment, the UG 442 Network transformer located in the 428 Massachusetts Avenue vault was inspected on 10/15/2014 and 4/20/2013. The following are the Asset Management inspection reports for both dates: 10/15/2014 Inspection Report

Note the fourth question “Vault – Network Transformer – Rust” and the corresponding response “Some Rust” plus the “View” link to the photo. The photo contains the hand written (cursive) comment shown above. This condition was not a contributor to the flashover as explained below.



4/20/2013 Inspection Report

During both inspections, the primary termination chamber was externally inspected using infrared technology to identify temperature irregularities. Temperature irregularity can be an early warning of a potential electrical insulation or connection problem. The “Vault – Check Transformer Infrared Termination Chamber” was “Normal” on both inspection dates.



4.0 Sequence of Events The following sequence of events was developed using information from IPL & IFD logs and records.

IPL removed and replaced the UG 442 Network transformer and protector at 428 Massachusetts Avenue. IPL inspected the adjacent equipment including opening the primary termination chamber on the UG 412 transformer located at 428 Massachusetts Avenue looking for damage or other indications of potential issues. No issues were identified.

Date Time (EDT) Remarks Status / AØ BØ CØ

3/16/2015 20:40 Pre-Event

UG 442 at 428 Mass Ave

Network Protector Closed

Protector Amp 400 A 405 A 380 A

Transformer Voltage 124 V 124 V 124 V

UG 442 Feeder Amps 125 A


3/16/2015 20:42 UG 442 Relayed

Post-Event

UG 442 at 428 Mass Ave

Network Protector Open

Protector Amp 0 A 0 A 0 A

Transformer Voltage 0 V 0 V 0 V



3/16/2015 20:45 IFD Log indicates they were dispatched - Cleared the scene at 21:43

3/16/2015 20:46 Sadtler receives text from Bentley

3/16/2015 20:46 IFD call is logged in OMS / reported as 430 Massachusetts Ave

3/16/2015 21:05 Bentley on Scene (Role - Incident Scene Commander)

3/16/2015 21:13 OMS shows IPL Truck 68 arrived at address

3/16/2015 21:43 IFD is cleared of the incident scene

Network Event on March 16, 2015

428 Massachusetts Avenue

Sequence of Events



5.0 Investigation and Analysis The investigation into the cause of the network event at 428 Massachusetts Avenue commenced as soon as the site was secured and safe. Prior to entering the vault, visual inspection of the bent primary termination chamber cover indicated an electrical flashover occurred inside the primary termination chamber on the UG 442 Network transformer. The Edison UG 442 was electrically isolated and grounds were placed so crews could safely enter the vault and perform the initial inspection. The crews took the following pictures to preserve evidence required for the Root Cause Analysis (“RCA”).

UG 442 Primary Termination Chamber (side view) Note: The bolt-on cover is bent and not attached at the bottom and sides.

UG 442 Primary Termination Chamber (bottom view)



UG 442 Primary Termination Chamber (front view) Note: The markings on the bolt-on cover indicate the termination chamber was retro-filled with FR3 insulting fluid on 5/29/2012 per IPL’s Response to: Independent Assessment of Indianapolis Power & Light’s Downtown Underground Networki. The benefit of retro-filling primary termination chamber with FR3 fluid is to prevent a sustained oil fire when an event occurs in the primary terminating chamber. The FR3 insulating fluid performed as designed by not sustaining combustion.

UG 442 Primary Termination Chamber (with cover removed) Note: The carbon markings in the upper right-hand corner (both pictures) and the hole in the insulation (right picture). The hole in the insulation is the result of the electrical flashover and occurred where the lead cable jacket stops and transitions to the copper wire mesh to form the electrical stress cone. The electrical stresses are the highest in this area.



Information from the Schweitzer SEL221Fii protection relay on the Edison Unit #2 power transformer/switchgear (which is the primary source for UG 442) was obtained to provide details about the fault, such as duration, magnitude, and phase involvement. Below is the oscillography obtained from the relay. Please note the time stamp for the SEL221F relay is off by 1 hour and 46 minutes (clocks are not synchronized) from the IPL transmission system time which remains EST throughout the year.

The upper portion of the graph shows the measured 3-phase 13,200 Volt current flow in Amps on the Edison #2 power transformer. Inspection of the graph indicates the pre-fault current was balanced between the three phases and was approximately 300 Amps per phase. During the fault event the “A” phase current peaked at approximately 6,000 Amps. The fault duration lasted approximately 4.5 cycles. One cycle represents 1/60 of a second (or 0.0167 seconds). The bottom portion of the graph shows the 13,200 Voltage phase to ground value which is nominally 7,630 Volts. The voltage graphs also indicate only “A” phase was involved and the source voltage at Edison Substation dropped by approximately 20% during the fault event and prior to the automatic relay isolation clearing of the UG 442 feeder breaker at Edison Substation. Based on the oscillography information the relay protection scheme and circuit breaker operated as designed. The Edison Substation UG 442 circuit protection relays detected



the fault and sent a trip signal to open the UG 442 feeder breaker in less than 0.1 second after the flashover occurred which isolated the fault.

5.1 Downtown Network Supervisory Control and Data Acquisition

(SCADA) The following charts provide both pre and post data from the March 16, 2015 event:

• Edison UG 412 – “B” phase feeder amps (3/16 00:00 to 3/16 24:00 EST) • Edison UG 442 – “B” phase feeder amps (3/16 00:00 to 3/16 24:00 EST) • 428 Massachusetts UG 442 – Secondary three phase current (3/16 00:00 to 3/16 24:00 EST) • 428 Massachusetts UG 442 – Transformer secondary average voltage (3/16 00:00 to 3/16 24:00

EST)

Edison UG 412 – “B” phase feeder amps (3/16 00:00 to 3/16 24:00 EST) – Items to note: Between the 18:00:00 and 20:00:00 marks on the graph pre-event current – approximately 85 amps; B phase current during the event spikes, post-event current approximately 104 amps.

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

Edison UG 412 B Phase Amps



Edison UG 442 – “B” phase feeder amps (3/16 00:00 to 3/16 24:00 EST) – Items to note: pre-event current – Approximately 125 amps; post-event current is 0 amps when the Edison Substation UG 442 feeder breaker opens. (See graph @ approximately 20:00:00 mark).

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

160.0

180.0

200.0

Edison UG 442 B Phase Amps



428 Massachusetts UG 442 – Secondary three phase amps (3/16 00:00 to 3/16 24:00 EST) – Items to note: the pre-event secondary 3-phase currents were approximately 380 to 405 amps, the post-event currents were all zero as a result of the opening of the network protector. (See graph between 18:00:00 and 20:00:00 marks).

0.0

100.0

200.0

300.0

400.0

500.0

428 Mass Ave - 442 Network Protector Amps

A Phase

B Phase

C Phase



428 Massachusetts UG 442 – Transformer secondary average voltage (3/16 00:00 to 3/16 24:00 EST) – Items to note: pre-event secondary average voltage is between 124 and 125 volts. Post-event the average voltage is zero as a result of the UG 442 Edison feeder breaker opening and the UG 442 network protector opening at 428 Massachusetts Avenue. (See graph between 18:00:00 and 20:00:00 marks).

50

60

70

80

90

100

110

120

130428 Mass Ave - 442 Network Trans. Volts



5.2 Primary Termination Chamber Observations Post Event The primary termination cover was found bent and attached to the housing via the upper bolts. The side bolts and threaded mounting studs at the bottom sheared off. This directed the release of the FR3 fluid and associated flash in a downward direction toward the floor of the vault. The FR3 fluid level post-event was even with the lower lip of the primary termination chamber. The remaining FR3 fluid was very black in color due to the carbon in the oil from the flash over. The copper mesh covering the stress cone on “A” phase cable came off during the event. The “A” phase copper conductor at the point of the flashover vaporized. The most probable cause is a breakdown of the insulation between the copper conductor and the copper wire mesh allowing the flashover to occur in an area of high electrical stresses. This area of high electrical stress is at the point where the cable insulation is exposed and the outer cable jacket is removed. A stress cone is installed to manage the electrical stresses between the primary copper conductor (which is at primary phase-to-ground voltage) and lead cable jacket and the copper wire mesh used in the stress cone. The flashover created carbon marks in the upper right hand corner of the termination chamber and on the inside of its cover. The location of the markings is consistent with the flashover on the cable. The cable enters the primary termination chamber through a wipe sleeve. IPL records show this transformer was placed in-service at 428 Massachusetts Avenue in June of 1980. The network transformer was manufactured by General Electric in 1954 at the Pittsfield, MA plant. It is important to understand the flashover occurred in the primary termination chamber; the transformer did not fail. The termination chamber connection was completed in 1980 when the unit was installed at 428 Massachusetts Avenue.

UG 442 Network Transformer Primary Termination Chamber as found

UG 442 Network Transformer Termination Chamber Stress Cones The two conductors on the left have copper mesh in place which has been wiped. On the right hand conductor note: 1) the copper mesh came off as a result of the flashover, 2) the



hole in the insulation and the varnished cambric tape in the upper portion of the stress cone, 3) copper conductor material missing (vaporized during the flashover)

UG 442 Network Transformer

(Left picture) Copper mesh material found in the bottom of the termination chamber; (Center picture) Shows the carbon marking inside the termination chamber relative to the location where the flashover occurred; (Right picture) Shows the carbon marking on the inside of the primary termination chamber cover.

6.0 Conclusions

Performance of the root cause analysis of the network event that occurred at 428 Massachusetts Avenue vault on March 16, 2015 has been completed and yields the following information. The fault occurred in the primary termination chamber of the UG 442 network transformer unit in the 428 Massachusetts Avenue vault and is a result of insulation dielectric breakdown in the “A” phase conductor in the upper portion of the stress cone. This created a phase to ground fault on the “A” phase conductor. Since the primary termination chamber was filled with FR3 fluid, which has lower volatility and higher flash point, the resulting flame associated with the fault was not sustained. Information collected from various monitoring systems confirms that all protection equipment operated as designed. Edison UG 442 circuit relay protection devices sensed the fault current and tripped open the circuit breaker in 4.5 cycles clearing the primary fault current. All 23 of the closed network protectors connected to the Edison UG 442 circuit operated as designed by opening. This prevented back feed from the low voltage side of the network transformers to the primary termination chamber flashover/fault location on UG 442 primary cable in the termination chamber at 428 Massachusetts Avenue vault. In summary, the impact of the incident was minor because the equipment worked as designed. Also, no customers lost power as result of the incident.



The insulation breakdown occurred in an area of high electrical stress. Exhibit 3 shows the stress concentration at the end of the outer conductive layer when stress control system is not applied. Exhibit 4 shows the traditional method of managing and reducing the electrical stress with the installation of additional insulating material in a cone shape and an outer conductive material to reduce the area of concentrated electrical stress. The goal of reducing the electrical stresses is to increase reliability of the termination.

Exhibit 3 – Uncontrolled Cable End – without Stress Cone

Exhibit 4 – Controlled Cable End – with Stress Cone



Instead of repairing the primary termination chamber cover and fasteners, IPL opted to change out the UG 442 Network transformer and protector at 428 Massachusetts Avenue. The primary cable connection on the new transformer is now accomplished through a bolted termination fitting. This style of connection is used when new transformers are installed as part of the commitment made by IPL in its January 2012 Response to: Independent Assessment of Indianapolis Power & Light’s Downtown Underground Networkiii .

7.0 Next Steps

• The following next steps are applicable to both Edison and Gardner Lane Substations since they serve the downtown network and have similar design and operating characteristics:

a. Investigate engineering methods which will result in a reduction of the

primary phase to ground fault current. The substation power transformers at Edison and Gardner Lane Substations and the impact to the secondary electrical characteristics should be evaluated and analyzed. By limiting the available phase to ground fault current the amount of energy released during the onset of an event will be significantly reduced.

b. Install digital relay protection on the Edison and Gardner Lane

Underground feeders. This will provide greater monitoring capability and data that can be used for planning and event analysis in the future.

• Remove one of the two remaining non faulted stress cones in the primary

termination chamber from the UG 442 network transformer that was removed from 428 Massachusetts Avenue and send to a third party for a laboratory analysis. The goal of this is to determine why the failure occurred causing the electrical flashover to occur.

• Pending the results of the laboratory analysis of the non-faulted stress termination, additional corrective actions may be warranted. The Company will make that determination, once the laboratory results have been received and reviewed.



i IPL’s Response to: Independent Assessment of Indianapolis Power & Light’s Downtown Underground Network: Recommendation 10b – “Explore the possibilities for fire retardant dielectric in vaults.” IPL retro-filled all network transformer primary termination chambers with the fire retardant dielectric FR3 fluid. http://www.in.gov/iurc/files/IPL_Response_Final_Complete_Version__011912.pdf ii SEL221F – Schweitzer Engineering Laboratories Distance Relay. IPL has installed the SEL221F relay on each of the three power transformers located at Edison Substation. The relays are installed to provide event capture information used for analysis. Such as the oscillography used in this report. iii IPL’s Response to: Independent Assessment of Indianapolis Power & Light’s Downtown Underground Network: Recommendation 3a – “Begin a program of retrofitting termination chambers with elbow fittings, and specifying such equipment for new or replacement network transformers.” In 2012 IPL retrofitted its existing stock inventory of network transformers with elbow fittings and the retrofitted equipment was available for deployment beginning in the 4th Quarter of 2012. IPL also modified its specification for the purchase of new transformers to require the elbow fitting. http://www.in.gov/iurc/files/IPL_Response_Final_Complete_Version__011912.pdf


root cause analysis for 428 massachusetts avenue network event on march 16, 2015 rca for...

Documents