Achieving High Performance through Leading Practices in Smart Grid Testing

How utilities can meet the testing challenges raised by smart grid deployments

2

Introduction New technology, new testing imperatives

A smart grid implementation program represents much more than a technology change for a utility. In fact, the effect of smart grid is to drive a radical end-to-end transformation of the way utilities generate data, present information, make decisions, execute processes and engage with their customers.

This pervasive impact is reflected in the business-critical area of testing. The sheer number and variety of components in a smart grid program create major challenges for conventional testing approaches, demanding new thinking that will help utilities avoid more complex and costly problems further along the utility’s smart grid journey.

With this in mind, the essential success factors for high performance in any smart grid implementation program include both: individual testing of the foundation elements/smart grid components, and collective testing of these elements and components in the context of the end-to-end business process flows. Utilities that conduct effective, coordinated testing at each of these levels are well-placed to realize the targeted returns and benefits from their smart grid investments.

The five major foundation elements/smart grid components are illustrated in the National Institute of Standards and Technology (NIST) diagram in Figure 1. They are:

1. Wide area networks (WAN), local area networks (LAN) and field area networks.2. Integration architecture.3. New IT applications.4. Legacy IT applications.5. Devices and hardware.

We will now examine these elements and components in more detail, highlighting the challenges and key testing considerations that the move to smart grid raises with each.

Elements and components

3

Distribution

Markets Operations Service providers

CustomerTransmissionBulk generation

RTO/ISO ops Transmission ops Distribution ops Third-party providerUtility provider

Figure 1. Foundation elements/smart grid components.

Retailer/wholesaler

Aggregator

Electric vehicle Distributed

generation

Distributedgeneration

Electric storage

Appliances

Thermostat

Customer equipment

Customer EMS

Energy market clearinghouse

Market services interface

Plant control system

Generators

ISO/RTO participant

EMS DMS

New IT applications

RTOSCADA

Metering system

Retail energy provider

Energy services interface

Devices andhardware

EMS

WAMS

Asset mgmt

Demand response

Transmission SCADA

Distribution SCADA

CIS

Billing

Others

Legacy IT applications

Billing

Aggregator

Substation controller

Datacollector

Substation device

Electric storage

Fielddevice

Home/building manager

WAN, LAN and field area networks

New IT applications Devices and hardwareLegacy IT applicationsIntegration architecture

Domain

Information network

Actor

Source: NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 1.0,” National Institute of Standards and Technology, U.S. Department of Commerce, January 2010, www.nist.gov/public_affairs/releases/upload/smartgrid_interoperability_final.pdf.

Domain gateway actor

Comms path

Comms path across owner/domain

Integration architecture

eBusinessInternet/

Wide area networks

Enterprisebus

Enterprisebus

WAN, LAN and field area networks

Internet/eBusiness

Substation LANs

Premisesnetworks

1

2

3

4

5

4

Foundation elements/smart grid components 1. Wide area networks (WAN), local area networks (LAN) and field area networks. 2. Integration architecture. 3. New IT applications. 4. Legacy IT applications. 5. Devices and hardware.

WAN, LAN and field area networks ChallengesThe key testing challenge that arises around WAN, LAN and field area net-works in a smart grid implementation is the need to incorporate existing infrastructure and configuration into the overall solution. In many cases, parts of the existing infrastructure must be rebuilt or replaced to support effective operation of the smart grid. This challenge is increased by the fact that legacy IT systems and field devices are often networked using a variety of communications protocols.

Key testing considerationsThe two most important consider-ations in testing smart grid WAN, LAN and field area networks are security and latency.

In terms of network security, it is vital to analyze and assess the effectiveness of the security supported by the existing infrastructure, conduct gap analyses of the target security standards, reconfigure the security capabilities where necessary and retest the security supported by the smart grid. An important consid-eration throughout is the fact that a virtual LAN is a critical component for implementing a truly secure smart grid.

In regards to network latency, the utility must fully test the levels of latency to minimize the impacts on business processes that support and enable transactions across multiple smart grid devices and applications.

After security and latency, the next two most important considerations are robustness and connectivity. A wide range of components—hardware, software, infrastructure manage-ment, transport medium and physical location—all contribute to the overall

robustness. The program must include testing focused on the redundancy, availability and disaster-recovery capa-bilities of these various components. Similarly, with connectivity, all criti-cal points in the network need to be tested for robustness and resiliency.

Time synchronization/propagation is a further important consideration that must be fully tested. The WAN/LAN/field area network must support the propagation of a common time reference, synchronized down to milliseconds for devices such as fault indicators. In terms of bandwidth, smart grids need sufficient capacity to transfer terabytes of data across strategic points on the WAN. Testing of data prioritization (quality of service, or QoS) is important to ensure data on critical events such as meter outages are given higher priority than noncritical data passing over the network, such as meter low battery alarms.

5

Integration architectureChallengesThe greatest challenge around the testing of integration architecture in a smart grid environment is the need for integrating a diverse range of applications—both legacy and new—based on varying platforms.

Key testing considerationsSecurity is a key consideration in testing the integration architecture. To effectively test security, all data elements, data tables, message schemas and enterprise service bus (ESB) transport layers need to be separate and isolated, alongside rigorous testing of tracing and data logging.

The service-oriented architecture (SOA) is a further important focus area, testing the services and interfaces between applications. As part of this interface testing, the program should verify transforma-tions, routing and data flow from an end-to-end perspective.

In combination with these considerations, business process management (BPM) should be tested to verify that existing business process functionality has not been unintentionally lost. For example, implementation of automated meter reading might have an impact on billing cycles within the billing system, and such impacts need to be tested, quantified and addressed.

Testing of the common information model (CIM) needs to focus on the minimum data definitions and attributes for systems within the program. This approach should use CIM to verify the data flowing out of systems and the proper processing of data coming in.

New IT applications ChallengesThe key challenge with new systems is that many new applications are not yet sufficiently mature to enable medium- or long-term certainty. New functionality is constantly being added, and the role of each application in the end-to-end process is still developing. Every new release requires significant regression testing, creating an ongoing overhead.

In many cases, data set-up needs to be implemented for new system perfor-mance testing as there is no existing data that can be leveraged for this purpose. Examples might include “last-grasp” signals from 100,000 meters in the event of a major power failure. Testing of new systems also creates challenges around training and skills, since a smart grid program requires resources with detailed knowledge of new applications. To support and enable effective testing, it is necessary to secure the commitment of vendors to train existing resources on the new applications.

Key testing considerationsClearly, the top consideration with new applications is whether they do their job. This means verifying the function-ality of the application and of any code that is affected by company- specific customization. A related requirement is performance testing to verify the ability of the application to meet the targeted integration stan-dards and performance requirements.

Error handling must also be tested to check that the new system’s routines for handling errors and excep-tions are working effectively, including logging of errors and escalation based on service level agreements. Testing the security of application involves verify-ing and ensuring that the application’s security complies with the specified security needs.

Legacy IT applications ChallengesFor performance testing of legacy systems, data set-up can be a chal-lenge if local privacy laws prevent the reuse of production data without first removing personally identifiable information. A further hurdle is that knowledge of legacy applications is typically held by the application users themselves. As a result, for testing involving the legacy application to be truly effective, a commitment to the testing program must first be secured among the application’s users.

Key testing considerationsEffective testing and verification of legacy applications’ error handling—including logging of errors and escalation based on service level agreements—is key. Also, additional application functionality will be needed to incorporate and integrate legacy applications into the smart grid program, so add-on functionality must be tested and existing func-tionality verified. To ensure that the systems meet the required integration performance requirements, legacy applications will have to undergo performance testing.

Devices and hardware ChallengesAt any one point of time in the field, multiple systems, devices and versions of device firmware will be present on the smart grid network. Data from all these diverse devices need to be processed by the IT applications. Also, each of the multiple versions of device firmware must be tested prior to deployment.

Key testing considerationsConnectivity is a major consideration, with critical points in the network needing to be tested for robust-ness and resiliency. Also, applications interfacing with the devices need to be tested for backward compatibil-ity. When device firmware is being upgraded, the testing procedure should follow the update process recommended by the vendor.

6

7

Key testing domains and challenges

Given the combination of diverse systems, hardware and internal and external networks in a smart grid implementation, it is imperative to ensure adequate test coverage in each of the testing phases and domains.

The key testing domains and challenges are: • Individual application functionality testing. • Physical hardware and network testing. • Security testing. • Performance testing. • Automated testing. • Regression testing. • End-to-end testing.

Individual application functionality testing The application testing phase ensures that the application performs in accordance with the functional requirements. It also verifies each application’s functionality, security, performance and capability to inte-grate with other applications within the smart grid program.

ChallengesNew smart grid applications such as advanced direct connect structure (ADCS) and meter data management systems (MDMS) are still evolving and tend to undergo major changes with every new release. This ongoing change adds a significant amount of reworking around application testing.

A further challenge, as previously noted, is that knowledge of legacy applications is often held by the application’s users themselves,

meaning resource constraints, skills and building buy-in from these users can arise as testing program issues. Data set-up for performance testing of individual applications can also be difficult due to the newness of sys-tems and programs and the lack of prior production data.

New systems are typically packaged applications and usually custom-ized to meet specific client needs, so the testing process brings with it an inherent risk of breaking the function-ality of the application involved. There is also a need for multiple environ-ments on smart grid implementations, reflecting the diversity of system, applications, devices and firmware.

Physical hardware and network testing Physical hardware testing verifies the functionality of different types of equipment. For example, meter test-ing needs to be performed per type of

meter rather than being standardized across the whole estate. In addition to various standard network tests, the testing of the networks will include penetration testing and will verify the handling of real-time loads, security and time synchronization across applications and the network.

ChallengesNetwork bandwidth and latency testing need to be performed under real-time loads, which are hard to reproduce in the testing environment. Partly as a result, environment avail-ability and the ability to link with the correct application environment are needed. Devices such as meters and in-home devices are highly dependent on embedded firmware, which should also be tested since they differ widely and may change regularly.

8

Security testing Security testing verifies that the data passing across the network is protected and used as intended. It must be performed on all com-ponents of the smart grid program, including new systems, legacy systems, interfaces, networks and hardware such as meters.

ChallengesUtilities are capturing customer data in higher volumes and greater detail than ever before, so it is becoming more urgent to test and ensure the secure storage, transmission and use of this information. Externally facing devices and hardware components are susceptible to attack and unauthorized access, so penetration and vulnerability testing needs to be conducted to identify potential risks and establish mitigating controls if necessary. Two-way communications enable both the “spoofing” of rate payers’ energy usage and of the energy provided back to the utility. These scenarios need to be taken into account during testing. Utility-owned meters have an expected life cycle of 10 to 15 years, so they need to be tested continu-ally to ensure they are still resilient. Also, these meters are often located at physically unsecured locations not under utility control, meaning that testing the physical safeguards on and within the meter is critical.

Furthermore, security testing must encompass software applications, as well as devices and physical hardware. Testing needs to take into consider-ation the overall system architecture with each system potentially imple-menting different security protocols and interoperability standards.

Performance scalability and stress testingPerformance, scalability and stress testing verify that the critical business processes can be executed across the entire solution at various load levels and under a range of circumstances from normal to extreme. This test phase also covers user interface response time.

ChallengesPerformance testing of the network will need the simulation of large volumes of physical hardware devices. Also, data set-up for performance testing of an end-to-end process is difficult to coordinate, as the data will traverse multiple applications, and large volumes of data need to be set up in each of the applications. Performance testing needs to be verified for multiple components—including application, interfaces and end-to-end processes. A further complication is that bottlenecks in one component can potentially impact entire business processes, making it important to identify interdependencies.

Automated testing Automated testing utilizes software scripts to test and verify functional-ity and effectively reduces the effort needed on subsequent test execution cycles. Testing of software applica-tions may be automated, but testing of hardware components such as in-home devices and meters is usually a manual process, unless meter or other device simulators can be used.

ChallengesThe major challenge with automation testing is the difficulty of controlling application development, with new smart grid systems typically subject to significant version updates over many releases. This means automated testing scripts from an earlier version will require constant maintenance and updating to ensure an effective

test set. Also, end-to-end automation is more sensitive and prone to failure, primarily due to the difficulties of setting up the data correctly across multiple applications.

Regression testingRegression testing verifies previously tested functionality and identifies defects that have been created inadvertently by the process of fixing other defects. It is treated as an activity within many of the other testing phases such as new applica-tion testing and integration testing.

ChallengesAs we previously noted, new IT applications around smart grids are still in the early stages of their evolution and experience major func-tionality changes between versions. As a result, integration regression potentially involves a complete retest of the applications in question.

Similarly, automated regression test scripts involving physical hardware and devices may suddenly become obsolete and invalid due to updates of device firmware, such as program identification changes for advanced metering infrastructure (AMI) meters. The challenges around regression test-ing are further increased by the fact that AMI is a “system of systems”—meaning changes in one system have ripple effects in other systems. This makes issue triage and isolation more complicated, especially during regres-sion testing.

End-to-end testingEnd-to-end testing verifies that all solution components integrate successfully with each other and with existing legacy systems to enable end-to-end business pro-cesses and meet functional/business requirements. End-to-end testing is conducted in controlled test environ-ments with production data, utilizing physical test devices such as smart meters and in-home devices.

9

ChallengesWith end-to-end testing, the principal challenge is the complexity created by the involvement of multiple systems, including legacy, vendor and packaged solutions. Data set-up for an end-to-end scenario will almost inevitably span multiple systems.

To overcome this challenge within an acceptable time frame, sequential releases of program functionality are often used to enable end-to-end testing to start earlier in the program. Also, thorough end-to-end regression testing is needed as new applica-tions will evolve and new application versions will be released during the course of the program.

End-to-end testing is further complicated by its dependency on physical hardware, raising issues around availability and the testing of functionality and performance.

There are also multiple application owners and a need to test multiple “hand-offs” between different systems. A thorough understanding of network latency is also needed, together with implementation of synchronous and asynchronous communications.

10

Mapping smart grid elements/components and testing domains

There is no direct one-to-one correlation between the challenges arising in the five smart grid foundation elements/components and the seven key testing domains. However, it is possible to draw a “heat map” of combinations of component and testing domains that are likely to give rise to the most significant challenges. Such a map is illustrated in Figure 2, with each intersection color-coded in red and tan. Red reflects the most challenging and tan less challenging.

Figure 2. “Heat map” of the challenges from smart grid components and testing domains.Testing domain/ foundation elementApplication functionality

Physical hardware and networks

Security

Performance

Automation

Regression

End-to-end

WAN/LAN networksIntegration architecture New IT applications Legacy IT applications Devices and hardware

Integration layer treated as application: Test transformations, routing and data flows.

New systems not mature enough. Constant update to product functionality with each new release. Test new system’s ability to handle multiple device firmware versions. New systems usually interface with devices, verify backward compatibility.

Verify new legacy functionality. Criticality of legacy system dependent on functionality being provided (billing, outage resolutions etc.).

Test devices’ capabilities to function on multiple firmware versions.

Test time synchronization, bandwidth, QoS, robustness, resiliency, connectivity and latency.

Device connectivity tested for robustness and resiliency of network.

VLAN is a critical component to implementing a secure smart grid. Penetration and vulnerability testing of the networks.

ESB transport layers security testing independent of applications. Calls to services with SQL injection, etc. Penetration and vulnerability testing of the integration layer.

Verify new system meets overall program security requirements. Penetration and vulnerability testing of the new systems. Smart grid require capturing large amounts of customer data. Data security testing.

Smart grid require capturing large amounts of customer data. Data security testing.

Test for spoofing of data from devices due to two-way communication.

Test networks and devices under real-time loads.

Use automation to test new system functionality over different releases.

Significant regression testing on every new application release.

Verify existing business processes are not broken due to addition of integrations between systems.

Other considerations• Performance test end-to-end transactions on the integrated solution.• Managing a testing effort on a smart grid program is difficult given the involvement of the different teams and many disparate systems. • Maintain strict entry and exit criteria when moving from one phase to another.

11

12

Leading practices and emerging trends for testing smart grid programs

Accenture’s experience working with utilities—and a wide range of smart grid implementations and testing programs—confirms that several leading practices are emerging, accompanied by related testing trends. These include:

Use of “meter simulators” and/or “device simulators” for application functionality testing and downstream application and interface performance testing. Implementation of a “test lab” for testing of physical hardware prior to installation in the field. The lab infrastructure is also used to test new firmware, devices and connectivity after a phased roll-out.

More thorough testing at unit level reflecting the need for a higher level of testing at the system level and requiring greater business knowledge.

Use of risk-based testing, through which the initial testing is focused on the most troublesome areas. This may dictate implementation of new systems or changes to existing revenue-generating business processes aimed at expositing defects earlier in the process or even during development.

Reliance on test-driven develop-ment, whereby development plans for interfaces and new systems follow the sequence of a business process to enable the reuse of data within a business process flow.

Use of SOA-specific testing tools such as HP Service Test, Service Tester for SOA Quality and iTKO LISA.

Reuse of test cases across different test phases such as stringing together application and interface test cases to generate end-to-end test scenarios and to test existing business process functionality using newly incorpo-rated systems. Implementation of a uniform, programwide “defect management life cycle” to support common understanding across application teams during defect triage.

Emphasis on quality from an end-to-end perspective.

Use of data and keyword-driven testing such as using custom code or macros to generate test cases from test scenarios and data permutations.

Increased use of methodologies such as Agile, along with the practice of involving actual application end users and subject matter experts in acceptance testing.

13

The “V-model”: promoting early testing and end-to-end focus on quality In our view, the most important leading practice is to test early to avoid significant (and costly) problems further down the line. The V-model (see Figure 3) provides a useful basis for testing. The model is a structured development framework, emphasizing quality all the way from the initial requirements stage to final testing. The V-model framework promotes stage containment by organizing the verification, validation and testing in and across human performance, technology and business processes throughout a project.

Busi

ness

Pro

cess

es

Busi

ness

pro

cess

es

Use

Cas

es &

Req

uire

men

ts

Use

cas

es a

nd re

quire

men

ts

Usa

bilit

y &

Pro

cess

Eff

icie

ncy

U

sabi

lity

and

proc

ess

effic

ienc

y

Stan

dard

s an

d ar

chite

ctur

e co

nsis

tenc

y

Relia

bilit

y an

d pe

rfor

man

ce

Exce

ptio

n ha

ndlin

g/re

cove

ry

Secu

rity

Application test

Integration test

Partial end-to-end tests

Full end-to-end testing

Busi

ness

cap

abili

ty b

luep

rint

driv

ers

Tech

nica

l arc

hite

ctur

e dr

iver

s

Cust

omer

Exp

erie

nce

& In

tera

ctio

ns

Cust

omer

exp

erie

nce

and

inte

ract

ions

Overall test approach

Component test Communications

Security

Performance/reliability User acceptance test

Operational readiness test

Plan Analyze Design Develop Test Implement

Use case finalization

Estimate and plan work

Requirements definition

High-level design

Detail design

Development

Figure 3. The “V-model” framework for smart grid testing programs.

14

Accenture’s methodology and approach

Logistics/infrastructure set-up Under Accenture Delivery Methodology, test labs are used to test devices in production-like environments, including executing real-world, end-to-end scenarios, testing firmware updates prior to roll-out into the field, and testing disruptive services such as meter connect/disconnect. In addition, using generators as part of the test lab process can generate phantom loads on meters.

Tools such as meter simulators and custom scripts help when device data—such as meter reads from a specific time period—can be difficult to reuse without additional data processing and/or rollback of processed data. Blended onshore/offshore testing teams provide cost advantages, enabling 24-hours-a-day/seven-days-a-week testing and fuller utilization of testing resources.

Creating a dedicated tactical security team can enable the execution of a broader range of security tests, while also ensuring that systems are tested by a party other than one doing the development. The dedicated security team should work in a tightly inte-grated way with the security test resources on the main test team. Careful planning of the deployment test reflects deep understanding of the risks of testing in production. If possible, testing should be conducted using devices from the test lab.

GovernanceThe stages of testing must be clearly defined, enabling the appropriate teams to become involved at each stage. Stringent entry/exit criteria are required for each stage of testing and for accepting code into the program environment. These processes include escalation processes for exceptions to the entry/exit criteria.

Backward compatibility should be tested through a specially implemented process to test for backward compatibility of newer versions of IT systems that might be released during the smart grid program. Business processes should also be reviewed periodically to identify changes from interim processes, the release of new capabilities or modified assumptions.

Test data needs to be planned proactively for each of the test phases, including cut-off data, mock conversion, manually generated and simulator data. Test management should include early definition and setting of expectations around the review criteria for test execution met-rics drawn from application testing performed by individual applications. In the vendor selection process, sat-isfactory performance testing results for new IT applications should be part of the vendor acceptance criteria.

Accenture’s proven approach to smart grid testing is called Accenture Delivery Methodology. Its key components include adherence to a “V-model” approach allowing for testing early in the life cycle, thereby avoiding critical defects being discovered at later stages. We will now examine Accenture Delivery Methodology under three headings: logistics/infrastructure set-up, governance, and test planning and execution.

15

Smart grid testing for high performance Accenture has a dedicated, industry-leading testing practice to help utilities achieve high performance in smart grid capabilities. We have some 12,000 testing professionals worldwide. They are supported by Accenture’s Testing Center of Excellence, which has locations in six countries and is part of the Accenture Global Delivery Network. Accenture also has alliance partnerships with major components vendors—devices, systems and net-works—and major testing vendors.

The Accenture Security group consists of a core group of more than 500 dedicated security specialists, plus more than 1,500 consultants with significant security experience. The team has deep experience and expertise in applying Accenture’s security framework and security

offerings in the context of utility, critical infrastructure protection (CIP) and specifically smart grid programs.

To learn how Accenture can help your utility optimize smart grid testing to achieve high performance with your smart grid implementation, please contact:

North AmericaSusheel Mutyalasusheel.mutyala@accenture.com

Asia PacificAnn Burnsann.v.burns@accenture.com

Europe, Middle East, Africa and Latin America (EALA) Maikel van Verseveldmaikel.van.verseveld@accenture.com

Automated testing should be applied prudently, following a data/keyword-driven approach. Accenture recommends the use of its internally developed Data Driven Automated Testing Approach (DATA), by consolidating leading practices and proven approaches at various clients.

Test planning and executionPerformance testing scripts designed and developed should be specific both to high-volume business processes, such as storm/feeder outage and restoration, and also the interplay between processes, such as 10,000 disconnects daily during mass deployment.

The project team should implement a clearly defined defect management process across all applications within the smart grid program. With this

management process in place, test execution uses a multipass approach to identify defects earlier in the test phase, retest fixes and regress as part of subsequent passes.

A modular framework should be incorporated for automation test scripting, reusing the applicable test harnesses and canned scripts from Accenture’s Testing Center of Excellence. Regression testing using automated scripts saves time and effort.

Accenture’s Testing Center of Excellence Accenture’s Testing Center of Excellence is a comprehensive and industrialized testing capability that clients can use for one or multiple stages of testing, whether for a specific application development initiative or across an entire organiza-tion. The Accenture Testing Center of Excellence combines dedicated test-ing resources, established processes and standard and reusable tools into a centralized testing service. Testing outcomes from Accenture are characterized by lower-cost, higher production quality and user accep-tance, on-schedule delivery and more flexibility to support business needs.

About Accenture Accenture is a global management consulting, technology services and outsourcing company, with approximately 211,000 people serving clients in more than 120 countries. Combining unparalleled experience, comprehensive capabilities across all industries and business functions, and extensive research on the world’s most successful companies, Accenture collaborates with clients to help them become high-performance businesses and governments. The company generated net revenues of US$21.6 billion for the fiscal year ended Aug. 31, 2010. Its home page is www.accenture.com.

Copyright © 2011 Accenture All rights reserved.

Accenture, its logo, and High Performance Delivered are trademarks of Accenture.

ACC7-1646/MOD-069

About the Accenture Utilities group The Accenture Utilities group has more than 30 years of experience working with electric, gas and water utilities worldwide. We work with 93 percent of the utilities on the 2010 Global Fortune 500 list, providing the deep industry knowledge, people and assets utilities need to develop the strategies and adopt solutions to improve performance in the dynamic energy market. With 100 smart grid projects in more than 20 countries, one of Accenture’s key focus areas is in helping our utilities clients with the transformation to a smarter grid. From generation to in-home energy management, from strategic blueprints to operational data analytics, and from the boardroom

to the operations center, Accenture offers the skills and experience that can help utilities frame their vision of a smarter grid and then achieve its many benefits.