UCA International Users GroupThe Quality Assurance Program

prepared by theUCA® International Users Group

Testing Subcommittee

UPDATED: 28 July 2008

Jack Robinson: UCA International Users Group SecretaryBruce Muschlitz: Chair Testing Subcommittee

Marco C. Janssen: editor of the QAP

International Users Group Mission

• Advancing interoperability for the utility enterprise by providing a global forum in which stakeholders can work cooperatively as members of a common organization to support international standards:

• Educational Activities• Improvement Process for International Standards• Testing and Certification• Promotional Activities

International Users Group Charter

• Not-for-Profit International Consortium

• Utilities and Vendors working together in support of international standards

• Over 109 Members From 26 Countries:• IEC TC 57 Members (Editors and Conveners of IEC 61850, CIM

and OpenAMI/DR)• Vendors of Control and Substation Automation Products • Utility Buyers of 61850, CIM and OpenAMI/DR Products• Testers and Certification Providers • Consultants, Universities, Research Facilities

Broad Technical Focus

• Utility enterprises are complex and cross many functional boundaries

• Promoting interoperability in the utility enterprise requires a similarly broad focus

• UCA IUG uniquely offers members an organization to promote standardization across several functional areas

UCA IUG – Technical Focus

• Communications– IEC 61850 – Substation Automation– IEC 61850 based extensions for wind, hydro, distributed generation,

etc.

• Application Integration– Common Information Model (CIM) IEC 61970 and IEC 61968– Generic Interface Definition (GID) IEC 61970

• Metering and Demand Response– OpenAMI – Information/data models, reference designs and

interoperability guidelines based on IntelliGrid and IEC standards.

ONE MEMBERSHIP

The Users Group andTesting

• The Users Group strongly supports testing

• Focus for testing is now on devices and systems developed in conformance with IEC Standards

• Users Group works closely with the IEC for problem resolution and tracking (e.g. WG 10 members are all in the IEC 61850 Technical Issues Users Group Subcommittee)

• Testing procedures, requirements and guidelines:– Quality Assurance of performed activities– Traceability of results

Testing Subcommittee

IEC 61850 Test Procedures

WG

OpenDRTest Procedures

WG

Testing TSC Org Chart

CIMTest Procedures

WG

• AEP A. Clark• EPRI J.Hughes Co-Chair• EnerNex B.Muschlitz Co-Chair• KCA J.Robinson Secretary• Triangle/ MicroWorks G.Schimmel • SISCO H.Falk • CIM Member• CIM Member• DR Member• DR Member

TestingSubcommittee

Testing SC

Committee / Working GroupEditors/ Representatives

• Testing Quality Assurance Program Editor: Marco Janssen (UTInnovation)

• CIM Document Editors

• Tester Accreditation Task Force (unique to each area)

• IEC 61850 Document Editors: – Tester System Accreditation:

Willem Strabbing (KEMA) – Server Product Test Procedures:

Richard Schimmel (KEMA)– Client Product Test Procedures: Richard Schimmel (KEMA)

Testing Web Site Support

• Testing Subcommittee / WG Areas (Meeting Agenda’s / Minutes, Draft Documents, Issues Discussion, Action Items, Tasks, Events)

• Member Only Testing Documents (Corporate Access)– Quality Assurance Program and Addendums– Product Test Procedures – Tester Accreditation Procedures

• Public Documents – QAP Overview/ Testing White Papers– Tester Certification (IEC 61850)– Testing Completed (IEC 61850) (Posted Certificates)

The Users Group andQuality Assurance for Testing

• A Quality Assurance Program (QAP) for IEC Product Testing and Test System Accreditation and Recognition is a major benefit to the members and is expected to save time and money for participants

• The QAP allows for the continuous improvement of test centers, test systems and the international standards

• The Users Group QAP identifies the means to quickly resolve IEC specification problems and, through IEC coordination, provides the vehicle for later update of IEC standards following IEC charter rules

Concepts of the QAP

• Feedback is essential– To keep all informed– To make sure standards are correct and well understood– To quickly resolve problems uncovered during testing

• Process is decentralized – All activities relating to testing are performed individually by parties

that may be other than the UCA® International Users Group– UCA® International Users Group plays a major role in coordinating

and managing information dissemination

• The process provides value for all who participate, and a strategy to bring all stakeholders together (users, vendors, testers, consultants).

QAP - The Development Process

IEC TC57WorkingGroups

ProductDevelopment

TestSystem

Development

TestLabs ProjectsTest Systems Tested Products Success

Specifications

Test CasesProducts

QAP - The Quality Assurance Process

IEC TC57WorkingGroups

Vendor

Manufacturer

System Integrator

TestSystem

Developers

TestLabs

Projects(Users)

All Parties Need FeedbackDifficult to Coordinate

QAP

Realistic Coordination

Standards ImprovementTechnical Issues (TISSUES)

TISSUES

Initiator emails

1st Proposal

“green” TISSUES “yellow” TISSUES

Final Proposal

100% approval

Resp. GoE member will create 1st proposal & distribute to GoE

After discussion a final proposal is distributed to TISSUES group for vote.

Selected and distributed to the responsible member of Group of Experts (GoE)

Could be a shortterm fix

IEC TC57 WG10

The Contents of the QAP

• Roles of:– Standards organization– Product development– Test cases, test system development, test systems and test

laboratories– Projects

• Rules, requirements and procedures for exchange of information between them– Input from and to:

• Vendors, manufacturers and system integrators for product development

• Test laboratories

• Vendors, manufacturers and system integrators for projects

• Users for projects

The contents of the QAP(cont'd)

• QAP Maintenance• Problem reporting and handling

– Standard problem report– Formal procedure for initial evaluation, proposals for resolution and

resolution and formal reporting of solutions and determine which developers are involved

– Short term recommendation to user(s) and vendor(s) on how to proceed– Process to upgrade the documents and/or test systems associated with

the problem

• Mechanism for identification, registration and publication of interoperability problems and the resolutions within various revisions/errata of IEC standards as well as specific revisions of the test systems and lab certified devices

The contents of the QAP(cont'd)

• Organization and infrastructure– Users Group Executive and Technical Committees– Audits

• Authority to organize and/or perform audits at testing organizations participating in the UCA® International Users Group QAP

• To verify the set up, implementation and use of the required quality system.• Periodically or upon request from a UCA® International Users Group

member with notice and mutually agreeable time schedule and agenda.

– Confidentiality of documents and information • In general testing results owned and controlled by testers and vendors• Results reported back to UCA® International Users Group and are

considered public information for dissemination to Members• Field problems carry the level of confidentiality as designated by the

reporting company

The contents of the QAP(cont'd)

• Organization and infrastructure– Information sharing

• Amongst UCA® International Users Group Members in accordance with the confidentiality guidelines

• Primary sharing vehicle is the Users Group web site under access and password control

• Public areas available to any one who visits the web site *• Confidential areas open to Users Group Members• Private Forum open to committee or working group members

* = www.ucaiug.mobi

The contents of the QAP(cont'd)

• Procedures, requirements and guidelines for the realization of tests including definitions for:

test configurations conformance test procedures / steps / resultsthe conformance test processstandard test procedure groupsIEC 61850 conformance blockstest case list test reporttest procedurescontrol of testing organizations and systems

planning of teststest definitions and set upformal and informal testingthe use of test toolsre-testingself testingprocurement stepstest architecture / system configurationtest architecture for conformancewhat is to be conformance tested?

The contents of the QAP(cont'd)

• Test Procedures

– Detailed test procedures for IEC 61850 Device Testing • Used by testing organizations

• Maintained by the UCA® International Users Group

• Contains Abstract Test Cases, based on IEC 61850 Part 10

– Certification done by test centers, users and test vendors(No direct certification of devices by Users Group)

– Any qualified facility may issue a “Product Certificate”– Users Group maintains certified devices list on the web site– Users / vendors that submitted the device for test own the detailed

test report / results

The contents of the QAP(cont'd)

• Control of Conformance and Interoperability

– Starts with accreditation and recognition of testing organizations

– No minimum conditions IEC 61850 protocol functionality to be supported in order to get a certificate

– Certificate remains valid as long as the owner follows the Users Group guidelines specified in the QAP

– Users Group maintains record of all tested and all tested and certified products and systems including the references to the documents and their respective versions that the tests performed were based upon

– Arbitrator role for the Testing Committee

The contents of the QAP(cont'd)

• Resolution of Problems resulting from Testing

– Testing must be documented, repeatable, and results comparable across tested equipment / applications

– QAP provides process for problem tracking, analysis, and resolution of tests, standards, and implementation agreements

– Users Group assists testing organizations in• Maintaining problem reports

• Resolving specification conflicts

• Modifying the standards as problems are uncovered

• Maintaining lists of certified products

• Ensuring the testing facilities keep up their qualifications per approved test procedures

The contents of the QAP(cont'd)

• Resolution of Problems resulting from Testing

– Access to QAP and problem information, along with key technical detail for users, product developers and integrators

– Status evaluation of own devices against a ‘non-aligned’ technical resource that presents an unbiased interpretation of the problems and possible resolutions

– Participation in the problem feedback process by Approved Testers – Information on problems uncovered in the field by vendors who

receive device certificates from approved testers– Vendors and testers shall include in their users and quality

assurance manuals, a problem identification, resolution and reporting process that includes the UCA® International Users Group feedback step

The contents of the QAP(cont'd)

• Improvement Process

– In accordance with the ISO 9000 series of standards, the UCA® International Users Group strives for continuous process improvement

– The community including members and non-members of the UCA® International Users Group are invited to use the mechanisms for providing feedback

– In accordance with the procedures laid down in the QAP the UCA® International Users Group will facilitate improvements.

Other Available Testing Documents

• Test Center Accreditation and Recognition Procedure for IEC 61850 Device Testing (Test System Accreditation)

• Conformance test for IEC 61850 Servers (Device Test Procedures)

• Testing of CIM Products (Draft by the CIM group)

• CIM Compliance Document/ Conformance Testing Requirements

• Conformance test for IEC 61850 Clients

Test System Accreditation

• Describes the procedure to qualify test centers

• Conditions that test centers should comply to in order to certify IEC 61850 based products and attach an Users Group label to the tested product

• To Date: There are 3 Users Group approved Test Centers for IEC 61850 Products: KEMA, Nari-Relays and ABB

• Tested device vendors (receiving certificates) participate in the Users Group Quality Assurance Program

Device Test Procedures

• Conformance test procedures and results in generaland in detail (IEC 61850 Example)

Testing Milestones• AEP/UG Testing Workshop (May 05: Demo of Testing

Capability, Tools, and Rollout of the QAP)• QAP Version 2.6 (March 2007)

– Update for General Quality Assurance Program– Addendum for IEC 61850 (June 2008) – Addendum for CIM and OpenAMI/DR (in preparation)

• Tester Certification IEC 61850– KEMA– ABB (Recognized)– Nari-Relays (Recognized)

• Device Testing IEC 61850 Server Procedures Rev 2.2 (October 2007)

• Testing Completed (IEC 61850) for 12 Vendors and 79 Products (posted on Web Site)

Testing Current Plans

• QAP Version 2.X– Addendum for CIM – Addendum for OpenAMI/DR

• Tester Certification IEC 6185• Device Testing IEC 61850 Server Procedures Use

Rev 2.2 (after October 2008)• Device Testing IEC 61850 Client Rev 1.0 (by early

2008)• Future Documents: Product Mark, Implementation

Agreements, Testing Specifications, Test Procedures for CIM and OpenAMI/DR

Bottom-Line Benefits

• Test and certification activities that lower costs for users

• Provide input to international standards activities at no additional cost and no travel

• Members only access to:– Resolution of technical issues related to standards– Industry experts via a global help desk– Technical documents and best practices from existing users– On-line discussion forums

• Discounts on paid seminars and exhibitions

Conclusions

• The available documents provide comprehensive means to assure the quality of the testing process

• Testing will be well documented, repeatable, and results shall be comparable across tested equipment/applications.

• The QAP provides an appropriate process through which to perform problem tracking, analysis, and resolution of tests, standards, and implementation agreements

• The UCA® International Users Group provides a vehicle for deployment and testing of high-quality products based on open standards for the Electric Utility Industry

To Join The UCA International Users Group

• Go To: www.ucaiug.mobi or E-mail kay@ucausersgroup.org or call (919) 847-2241

• For Public Documents and Information: www.ucaiug.mobi or E-mail bruce@ucausersgroup.org or jack@ucausersgroup.org