© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
ERIGrid Holistic validation procedure
and test specification
Kai Heussen
Center for Electric Energy - Department for Electrical Engineering
Technical University of Denmark
Workshop “Designing and Validating the Future Intelligent, Electric Power Systems”
September 6, 2017, Fuldatal/Kassel, Germany
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Coordinated
Voltage Control
14.09.2017 2
Validate this!
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
HOLISTIC SYSTEM VALIDATION
The Vision
14.09.2017 3
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
“Holistic” System Validation
I. System validation: alignment of Specifications & Testing
II. Integrated hardware & software testing Validate “systems” not components.
III. Tests that combine multiple domains e.g. Power, Comm. and Automation
IV. Systematically design tests & integrate results from various experiments for a holistic assessment i.e. combine simulation, co-simulation, HiL, PHiL, CHIL, different Labs, ...
14.09.2017 4
Specification | Testing
&Debugging
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
System validation - a holistic procedure
14.09.2017 5
"Holistic testing is the process and methodology for the evaluation of a concrete function, system or component (object under investigation) within its relevant operational context (system under test), corresponding to the purpose of investigation”
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
System Configuration - Defines components of a system - Includes: Object of investigation
Use Case - Defines functions of a system - Requirements define Test Criteria
A generic experiment / validation
14.09.2017 6
Purpose of Investigation - Experimental design - Test procedure
Test/ Experiment
Target Input
Configuration
Object of investigation Experimental setup
Boundaries of experiment Testing Tools
Data Exchange
Test Criteria Performance
indicators/ Test metrics
controllable & uncontrollable
input parameters
Test Case
Quality of experiment: error type; uncertainty; quantification of error
experiment assessment:
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
‘Design of Experiments’
efficient test design due to sampling-oriented approach
14.09.2017 7
Choose samples
Test/ Experiment
Target Input Params
Target measures / metrics e.g. “average control error”
design sampling space on
a ‘need-to-know’ basis e.g. 3 levels of package loss
rate, 20 levels of disturbance, …
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Holistic Test vs. Component Test
Component Test
e.g. Inverter MPPT test, anti-islanding
No interactions with the system
Usually open loop test (predefined
voltage, frequency etc setpoints are
applied to the DuT)
Holistic/System Test
Combining several tests
(testing
process)
Using
simulations
Testing a system rather than just
component
9/14/2017 Tutorial 8 Images: Panos Kotsampopoulos (NTUA); DoA
DC AC
PV simulator AC gid simulator Hardware inverter under
test
RTDS Simulated Network
PV inverter PV simulator
Central Controller (optimization)
PHIL
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
HOLISTIC TEST DESCRIPTION
The basics
14.09.2017 9
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Holistic test description
16/06/201
7 ERIGrid 3rd General Assembly - NA5 status
1
0
THR
EE le
vels
of
spec
ific
ati
on
Map/split
Test Case (TC)
Use Cases Test Objective(s)
Experiment Specification
Map/split
Experiment Specification
Experiment Specification (ES)
Test specification Test specification Test Specification (TS)
Generic
Specific
Lab
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Holistic testing procedure –
different mapping steps
16/06/2017 ERIGrid 3rd General Assembly - NA5 status 12
5
6
4
3
2
1
RI 1 RI 2 RI n
Experiment
Specification
Exp. Spec.
Experiment
Specification
Exchange of data and results
Holistic Test Case
Sub-test specification
Sub- test
spec. 2
Sub-test specification n
Division into individual (sub-)tests
1
3
5
6
7
Experiment
in RI n
Exp.
in RI 1 …
…
…
Exp.
in RI 2
Use Case(s) Use Case(s)
holistic
test
evaluation
test
refinement
res
earc
h in
fra
stru
ctu
re (R
I) c
ap
ab
ility
pro
file
s Scenario & Generic System Configuration
Use Cases
2
4
Test Objective Mapping from SC, UC,
TO to holistic test case
Mapping of holistic test case to (sub-) test specifications, i.e. specific (sub-) test
system(s)
Mapping of (sub-)tests to RI and specify
experiments
Mapping between tests resp. experiments
Mapping (preliminary) results to test adjustments
Mapping of results of experiments to
represent holistic result
x – mapping step
5 3 2
x Process steps
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Validate this! WHY TO TEST?
WHAT TO TEST?
WHAT TO TEST FOR?
HOW TO TEST?
Key Questions to be answered for test specification:
14/12/2016 External presentation 13
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Validate this! WHY TO TEST?
WHAT TO TEST?
WHAT TO TEST FOR?
HOW TO TEST?
Key Questions to be answered for test specification:
14/12/2016 External presentation 14
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Test System & Domain
14/12/2016 External presentation 15
System under Test (SuT):
is a system configuration that includes all
relevant properties, interactions and
behaviors (closed loop I/O and electrical
coupling), that are required for evaluating
an OuI as specified by the test criteria.
Object under Investigation (OuI): the
component(s) (1..n) that are subject to the
test objective(s).
Remark: OuI is a subset of the SuT.
Domain under Investigation (DuI):
Identifies the domains of test parameters
and connectivity relevant to the test
objectives.
OuI
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
SuT
measurements
Test System Functions
14/12/2016 External presentation 16
Functions under Test (FuT): the functions relevant to the
operation of the system under test, as referenced by use
cases.
Function(s) under Investigation (FuI): the referenced
specification of a function realized (operation-alized) by the
object under investigation. Remark: the FuI are a subset of the FuT.
OuI
DER P,Q control
optimization in the controller
state estimation
OLTC tap control
measurements measurements
Use Cases define Functions Of Systems (IEC 62559 approach)
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
WHY TO TEST?
WHAT TO TEST?
WHAT TO TEST FOR?
HOW TO TEST?
Key Questions to be answered for test specification:
14/12/2016 External presentation 17
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Purpose of Investigation (PoI)
o Verification
o Validation
o Characterization
14/12/2016 External presentation 18
Scoring / Performance
Modeling / Understanding
Test objectives/PoI: Characterization and validation of the DMS controller
1. Convergence of the optimization (validation)
2. Performance of the optimization under realistic conditions (characterization)
3. Accuracy of the state estimation (characterization)
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Designing Test Criteria Detailing Sequence
Test objective PoI Test Crit.
Test criteria: How to break down
the PoIs?
– Target metrics (criteria): list of
metrics to quantify each PoI
– Variability attributes:
controllable or uncontrollable
parameters to “disturb” SuT
– Quality attributes (thresholds):
test result level or quality of the
TM required to pass or
conclude the testing.
14/12/2016 External presentation 19
Target metrics: 1. 1.1 convergence (when/how
often?), 1.2. How fast?, 1.3. solution quality
2. 2.1 Voltage deviation 2.2 number of tap changes, 2.3 network losses
3. Voltage, P, Q estimation errors Variability attributes: Load patterns (realistic, annual variation; applies to criteria 1-3); Communication attributes (packet loss, delays) Quality attributes (thresholds):
“1.2: convergence within 2 sec” (validation) “3.* estimation quality characterized with confidence 95%” …
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
WHY TO TEST?
WHAT TO TEST?
WHAT TO TEST FOR?
HOW TO TEST?
Key Questions to be answered for test specification:
14/12/2016 External presentation 20
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Test Specification & Design
Given:
Purpose of Investigation (PoI) & Test Criteria
System & Domain categories and relations
To Specify:
Precise system (specific system configuration)
Which variables to manipulate & which to measure
How to quantify the test metrics (based on test data)
Sampling of the input spaces (design of
experiments methodology)
Combination and interpretation of the outputs
The test design / procedure.
Mapping to actual lab setup (experiement setup)
14/12/2016 External presentation 21
Research
Infrastructure (RI)
Experiment Specification
Test Case SuT, FuT, PoI, TC
Test Specification Test Design, I/O
1
6
7
Experiment in RI
test
evaluation
test
refinement
TO
res
earc
h in
fra
stru
ctu
re (R
I) c
ap
ab
ility
pro
file
s
SC UC
2
4 map
3
5
Test Specification Test Design, I/O
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Scoping & specification of
test system.
Separate specification
of lab implementation
Detailing test setup &
Mapping to the Lab
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
FIRST APPLICATION EXPERIENCE
14.09.2017 23
Can I just say, that it's very nice to get these
questions sorted out now, rather than when
you're sitting down and have to implement
something.
You would then usually go "Oh shit, how does
this work again?“ – ERIGrid participant
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Collaboration with ELECTRA
ELECTRA – Web-of-Cells (WoC) concept
– large set of Use Cases: distributed control
– concurrent development & lab
implementation
Challenge: how to track & convince that
ongoing experiments actually “validate” the
ELECTRA WoC
Result (intermediate):
– Gap analysis based on Test Criteria &
System configuration vs. ELECTRA goals
– organization of test case clusters;
collaborative design of test formulation
14.09.2017 24
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
ERIGrid Transnational Access:
Preparation & Documentation
External Lab users apply description procedure
E.g. DiNODR – distribution network oriented application of demand response – currently ongoing in SYSLAB
“The preparation work helped us a lot. Except minor changes in the plan and configurations due to a number of device, communication and control unavailabilities, we are following our test and experiment specifications. The template is also useful for our user team to exchange ideas in an organized and effective way. “ - Alparslan Zehir (DiNODR)
14.09.2017 25
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Conclusions & Future work
A clear vision for “holistic validation”
First results:
– 3-level - Test Description template & guidelines
– Multi-Domain System Configuration description (CIM compatible)
– Several successful applications & encouraging feedback
Future work:
– Further exemplify, simplify & detail description method
– Develop & apply full holistic validation procedure
14.09.2017 26
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Thank you for your attention!
Get your copy of the ERIGrid Test Description here.
14.09.2017 27
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Research Infrastructure (RI)
Experiment Specification Experiment Design, Experiment setup
Test Case System under test, Object under Investigation, Functions,
Purpose of Investigation, Test Criteria
Test Specification Test Design, Test System Configuration, Input & Output
map to specific test systems
1
6
7
Experiment in RI
test evaluation
test refinement
Test Objective
res
earc
h in
fra
stru
ctu
re (
RI)
ca
pa
bili
ty p
rofi
les Generic System
Configuration Use Cases
2
4 map to concrete lab setup
3
5
“pro
fess
ion
al”
seq
uen
ce
“learner” Start
anywhere. Build
specification iteratively.
Overall Specification & mapping procedure
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
IEC TR 61850-7-6 Guideline for definition of Basic Application Profiles
(BAPs) using IEC 61850
3.1 Compliance Accordance of the whole implementation with specified requirements or standards. However, some requirements in the specified standards may not be implemented. [SOURCE: CEN-CENELEC-ETSI SG-CG Report on Interoperability CEN_9762_CLC_9624 – clause 12.1 Terms and definitions]
3.2 Conformance Accordance of the implementation of a product, process or service with all specified requirements or standards. Additional features to those in the requirements / standards may be included. [SOURCE: CEN-CENELEC-ETSI SG-CG Report on Interoperability CEN_9762_CLC_9624 – clause 12.1 Terms and definitions]
3.3 Conformance test check of data flow on communication channels in accordance with the standard conditions concerning access organization, formats and bit sequences, time synchronization, timing, signal form & level and reaction to errors. The conformance test can be carried out and certified to the standard or to specifically described parts of the standard. The conformance test should be carried out by an ISO 9001 certified organisation or system integrator. [SOURCE: IEC 61850-4]
14/12/2016 External presentation 33
Validation
Verification
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Holistic Test Case Example
TEST CASE:
Narrative: For a DMS controller in development stage (simple implementation) the performance of the DMS algorithm and controller should be evaluated under realistic conditions. This test, could be seen as the last step before installing the DMS in the field.
SuT: DMS, DER, OLTC, transformer, distribution lines, telecom network
– OuIs: DMS_controller
– DuI: Electric power and ICT
FuT: DER P,Q control, measurements, OLTC tap control, comm. via ICT
– FuI: optimization in the controller, state estimation
Test objectives/PoI: Characterization and validation of the DMS controller
1. Convergence of the optimization (validation)
2. Performance of the optimization under realistic conditions (characterization)
3. Accuracy of the state estimation (characterization)
Test criteria – how to formulate these objectives?
Potential Test setups: • Pure simulation (e.g. co-simulation) • Combination of virtual & physical interfaces and simulated components. PHIL and CHIL • Full hardware setup
Target criteria - Variability attributes: - Quality attributes
© The ERIGrid Consortium
EU H2020 Programme GA No. 654113
Holistic Test Case Example
TEST CASE: SuT: … OuIs: DMS_controller; DuI: Electric power and ICT FuT: FuI: optimization in the controller, state estimation Test objectives/PoI: Characterization and validation of the DMS
controller 1. Convergence of the optimization (validation) 2. Performance of the optimization under realistic conditions
(characterization) 3. Accuracy of the state estimation (characterization)
[ ] Test criteria – how to quantify these objectives? Target criteria: 1. 1. convergence (when/how often?), 2. How fast?, 3. solutions
quality (how suboptimal etc.?) 2. Voltage deviation of all the nodes from 1 pu, number of tap
changes, network losses 3. Voltage, P, Q estimation errors
Variability attributes: Load patterns (realistic, annual variation; applies to criteria 1-3); Communication attributes (packet loss, delays) Quality attributes (thresholds):
“1.2: convergence within 2 sec” (validation) “3.* estimation quality characterized with confidence 95%” …