super computing with powerworld simulator
TRANSCRIPT
Super Computing with PowerWorld SimulatorTracy RolstadFebruary 2016PowerWorld Client ConferenceAustin, TX
Education
⢠Tracy Rolstadâ Diploma, Naval War College, College of Naval
Command and Staffâ BSEE, University of Idahoâ Nuclear Navy
⢠Nuclear Operational Prototype (S1C)⢠Nuclear Power School (Reactor Operator)⢠Electronics Technician School
â Radar, Communications, etc
ResumeâŚ
â Avista Corporation⢠Senior Pwr Sys Consultant, System Planning⢠WECC TSS Chair, Vice Chair, Secretary (former all)
â Utility System Efficiencies⢠Senior Power Systems Analyst
â The Bonneville Power Administration⢠Senior Engineer, System Operations
â The Joint Warfare Analysis Center⢠EP Senior Analyst, PACOM Chief of Targets⢠Special Technical Operations Action Officer
â Nuclear Navy (Attack Submarines)⢠Chief Petty Officer (ETC/SS)⢠Engineering Watch Supervisor
4
The Cray Y-MP: eight 32-bit processors capable of 333 MegaFlops each. Combined, the Cray Y-MP could sustain a speed of over 2 GFlops. The CPUs ran at a blazing 167Mhz and could process both 24-bit and 32-bit instructions. It cost ~$20 million in 1995.
Fastest processor for a desktop (~1995) was a 486DX/2 66. System bus of 33Mhz and an internal clock rate of 66Mhz. It was a 32-bit processor capable of 2.67 MegaFlops. It cost $4,000.
Lenovo T440 (i7-4600, 2 cores @2.10 GHZ). ~3 GFlops range (laptop with 1 CPU)⢠The i7 has ~1.3 billion transistors â˘Running a 64 bit OS (with far too much x86 software)⢠Costs about $700
Avista is edging towards âteraFloppingââŚbut practically Flops are a meaningless measure. We use a double Palo Verde generation trip as a measure (i.e.2PVâŚseconds per 2PV simulation)
5
PowerWorldâs Distributed Computing
Host
RemoteRemote
Remote
Remote
Remote
Remote
RemoteRemote
Remote
Remote
Remote
Remote
Remote
Remote ⢠Uses all CPU cores on your PC for analysis
⢠Can use all CPU cores on networked PCs as well
⢠The helpful folks in Avistaâs technology group got us 14 dedicated PCs
â 176 cores of analysis power⢠It rocks!
â We have our own LAN⢠DCOM on corporate LANâs is not
desired⢠Virus 101 says no DCOM⢠IT is happy with us being separate
5AvatarâAvista Advanced Transmission & Reliability
6
The Avista âbackboneâ
A âSupercomputerâ is BoringâŚ
It is also inexpensive and easy to build
What is this?
Avistaâs âSuper Computerâ
⢠5 Lenovo P700 (x2 CPU) w/32 GB RAMâ Total of 10 CPU (2 cpu, 12 core each) for 120 cores
⢠Server class machines (#6 on Intel speed list, ~$10K each)
⢠8 HP Z400 (x1 CPU) w/16 GB RAMâ Total of 8 CPU (6 cores each) for 48 cores
⢠1 HP Z800 (x1 CPU) w/16 GB RAMâ Total of 1 CPU (8 cores)
⢠Total Core Count is ~176â Count varies as we refresh machines or take them
out of the clusterâ More is better/fast is better. Go big or go home! â Donât be cheap. Donât use junk!
Passmark CPU Benchmark
6
112
311
505
P700
Z400
Z800
T440
Buy fast computers. Patches matter. Fast computers hide the need for patches!⢠Laptop-T440 does 2 PV 10 second sim in 5 min, 6 sec (have seen 14 min)
⢠Jamie can talk about the patches and their possible effects⢠Desktop-P700 does 2 PV 10 second sim in 3 min, 56 sec
Double Palo Verde (x 1000)
Double Palo Verde Trip
⢠The double Palo Verde Generator Tripâ Well known WECC outage and test
⢠Loss of ~1376 MW x2 (total loss is 2752 MW)⢠Excellent case checking contingency
â It HAS happened in the pastÂť 14 June 2004 all 3 Palo Verde units were lost
Note the Data Issues (it never ends)
Double Palo Verde
⢠Is computationally âbusyâ⢠Is an excellent benchmark for WECC
â Stability is a one at a time process â Power flow has queuing that you set up (later slides)
⢠Timing Tests (Almost irrelevant at this point)â Single ten sec 2 PV sims on P700 = 4 minutesâ 1000 ten sec 2 PV sims on cluster = 60 minutes
⢠We observe overhead when âassembling resultsââ 1000 * 10 = 10,000 sec or 167 minutes of sim time
⢠âFaster than real timeâ in aggregateâ Saved an FTEâŚwent from days to minutes
Google: 4000 minutes is how many days?
Avistaâs present transient stability list is ~1000 contingencies.⢠TPL-01-04 is moving this number much higher.
We look at as many as 20 cases in a year for compliance purposes.⢠Our studies took months before distributed computing (4 min per TS sim).⢠Software begat hardware.⢠Software & Hardware equals Good!⢠More is betterâŚto a point which we have not found yet
Engineering judgment for N-1-1âŚâ˘ Simulation = goodâŚjudgment = not as good as you think!
W i d b N 1 1 ff t
Things Donât Always Work (4168 hung)
Reboot and Patch Software
Distributed Computing Setup
Watching it assign 176 problemsâŚ
Assigns176 ctgsto 176 coresâŚThen you wait
Processing the results (from 1000 2 PV)
Results come back to a temp directory
Watching a machine work
Looking at machine adequacy
Look at speed,memory, cores, etc
Final Results (months turn into hours)
⢠1000 double Palo Verdes in 60 minutesâ 60 minutes of actual computing for 167 minutes of
simulated event time.⢠âFasterâ than real time in aggregate
⢠90,000 power flow contingencies in 18 minutesâ Yes, when being rigorous and complete you run this
manyâ Our experience is that engineering judgment is poor
judgmentâŚsimulation removes doubt⢠Cost $100KâŚBenefit of at least one FTE
ongoingâ We see 2 FTE emerging with more PCâs
Why such volume of simulation?
Future Steps
⢠Dashboard with computer management toolsâ c:\Windows\System32\taskkill.exe /F /IM
pwrworld.exe⢠Fault tolerance for computer failure or hanging⢠Speed increase opportunities
â Optimization, code grooming, elegance, efficiency, etc⢠Cloud Computing (new PWC service perhaps)⢠More people doing this!
â Avista is out in front here and would welcome others⢠Thousands of sims debugs PWS quickly⢠Bye bye x86âŚhello 64 bit PWS
Questions
⢠Questions, thoughts, ideasâŚâ˘ Food for thought
â How fast would ERCOT studyâs take?â Eastern Interconnection?â State Estimated case work for RCsâ 64 bit OS and the future of simulation
⢠Calculate the RAM limitâ exabytesâ What does that even look like