the soft grid 2013-2020: big data & utility analytics for ... · pdf filethe soft grid...

A Greentech Media Company

THE SOFT GRID 2013-2020: Big Data & Utility Analytics for Smart Grid

Research Excerpt

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT

GTMRESEARCH APRIL 2013 2©2013, GREENTECH MEDIA INC ALL RIGHTS RESERVED

RESEARCH EXCERPT

Research Excerpt

This is an excerpt from the December 2012 GTM Research report “The Soft Grid 2013-2020: Big Data and Utility Analytics for Smart Grid”. Research for this report was conducted over a six-month span and included primary and secondary research as well as extensive interviews with both industry players and utilities.

GTM Research, a division of Greentech Media, provides critical and timely market analysis in the form of research reports, data services, advisory services and strategic consulting. GTM Research’s analysis also underpins Greentech Media’s webinars and live events.

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT Executive Summary And Key Findings


1. EXECUTIVE SUMMARY AND KEY FINDINGS

Analytics: Key Findings

1. While utilities like to claim that they have analytics, they really don’t. Utilities tend to have last-gen business

intelligence (BI) reporting solutions that they call “analytics,” but that typically amount to not much more than

reporting tools or descriptive analytics (primarily based on older database architectures running SQL), as

opposed to the real-time and predictive analytics using complex event processing to which the term “analytics”

is now commonly understood to refer.

2. Utilities are now seeking to become more proactive in decision-making, adjusting their strategies based on

reasonable predictive views into the future, thus allowing them to side-step problems and capitalize on the

smart grid technologies that are now being deployed at scale. Predictive analytics, capable of managing

intermittent loads, renewables, rapidly changing weather patterns and other grid conditions, represent the

ultimate goal for smart grid capabilities.

3. In this report, we present a taxonomy that identifies the three major domains in which analytics can aid utilities,

all of which are ripe with opportunity.

A. Enterprise analytics

B. Grid operations analytics

C. Consumer analytics

FiguRE 1-1: uTiLiTiES’ THREE PRiMARY DOMAiNS FOR ANALYTiCS

SOuRCE: gTM RESEARCH

GENERATION

UTILITY INFRASTRUCTURE CONSUMER

TRANSMISSION SUBSTATION DISTRIBUTION HOME / BUILDING / DATA CENTERS

DISTRIBUTED GENERATION AND STORAGE

POWER LAYER INFRASTRUCTURE

COMMUNICATION LAYER END-TO-END COMMS PLATFORM

• Moving from Traditional, Historical Analytics

to Real-Time Predictive Analytics

• Complete Situational Awareness

• Business Intelligence (BI)

• Trading with “live look” at the Grid

Simulation/Visualization

ENTERPRISE ANALYTICS CONSUMER ANALYTICS

• Behavioral Analytics

• Tiered Pricing - Trading, Selling Megawatts (DR)

• Building Energy Management

• Power Analytics (Load Flow)

• Social Media Data Intergration

• DG/EV/Microgrid Analytics

GRID OPERATIONS ANALYTICS

Grid Optimization and Operational Intelligence

• Asset Management Analytics

• Crisis Management Analytics

• DMS Analytics

• Outage Management Analytics/Fault

Detection and Correction

• Weather/Location data

•Mobile Workforce Management

•Energy Theft

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT Executive Summary And Key Findings


4. It is our prediction that in three years, talking about analytics without mentioning big data will be a bit like

talking about email without mentioning the internet - - the two will become intrinsically linked, with one being an

application (analytics) sitting on top of the other, the foundational layer (big data storage and processing).

5. GTM Research believes that it is “high-performance” analytics, such as predictive analytics, which will prove to

be the most significant value-add in the big data age, as new data management technologies prove reliable and

fundamental, and as data storage infrastructure moves to commoditization.

Utilities’ Limited Experience With Analytics: Key Findings

1. Based on discussions with utility CIOs, the utility industry appears to be weary of the process of selecting and

commissioning custom products from vendors and of the consultant-heavy experience of deploying them; open-

source big data products offer a future with more flexibility and lower costs.

2. The four biggest challenges for utilities in terms of having enterprise IT architectures sufficiently prepared for

smart grid and big data are:

A. Siloed systems that hinder easy data sharing

B. Systems integrations is no small task

C. No existing platform in place for unstructured data

D. No single platform is going to be able to handle all needs

All of these obstacles speak to the central challenge of making disparate, incompatible datasets usable and valuable

across the enterprise.

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT The Emergence Of The Soft Grid


2. THE EMERGENCE OF THE SOFT GRID

2.1 Utilities’ Existing and Evolving IT Architecture Challenges

It is necessary to stress that utility IT architectures are in many ways only the jump-off point when it comes to realizing all of the benefits that can be reaped via big data and analytics. In other words, smart IT architecture has to be viewed as the gateway to a smart grid. An informal survey of utility CIO and CTOs that GTM Research conducted confirms that very few utilities have an official, overarching data strategy in place today.

The deployment of “smart hardware,” including smart meters and distribution devices such as automatic voltage regulators, is not only continuing, but will also accelerate as these devices become even more affordable. However, in order to leverage the capabilities of these new devices and to implement smart grid capabilities like dynamic pricing, grid optimization, self-healing grids and renewables integration, utilities desperately need to turn their attention to upgrading their IT systems and architecture. To the industry’s credit, many utilities are currently engaged in this process.

As they do so, many are discovering that they have an out-of-date patchwork of legacy systems with little, if any, architectural consistency. In the past, ad hoc point-to-point integration between pairs of applications was sufficient to handle basic needs, such as entering outage reports from customer service applications into an outage management system, or creating an engineering work order for execution by a maintenance crew using a mobile workforce management application. However, the age of big data could have devastating results on utility systems if IT architectures are not sufficiently designed and engineered for the level of performance and sophistication it will require.

Over the past three to five years, utility executives have been discovering that the ad hoc and unplanned nature of their systems threatens to block their forward progress in achieving smart grid business goals – and frankly this realization came before it was evident that big data was on the way! Every utility now needs a pragmatic roadmap that delivers on the promise of smart grid by leveraging and integrating legacy systems too complex to replace (over the immediate term), while putting a comprehensive plan in place to account for big data, as well as overcome the four biggest data challenges utilities are facing (see below).



FiguRE 2-1: FOuR MAJOR iT ARCHiTECTuRE CHALLENgES FOR uTiLiTiES

1. Siloed systems that prohibit easy data sharing. Many smart grid applications are composite applications

that draw on data and functions from multiple systems.

2. Systems integration is no small task. Related to the challenge of siloed systems is the challenge of creating

the underlying architectures that allow easy data access, sharing, and collaboration between systems. It

is particularly difficult to upgrade architectures that serve as the foundation for electric grids on which

millions of customers depend.

3. No existing platform in place for unstructured data. An estimated 75% to 90% of all new data being

generated is unstructured. Utilities as a group are ill prepared for this shift, and most have not explored or

tested big data platforms in a meaningful way.

4. No single platform is going to be able to handle all needs. Companies like Facebook and Twitter have had

to constantly rebuild and update their architectures in order to meet their ever-evolving, rapidly expanding

needs. This experience likely will be applicable to the utility space, as well. Utilities must look to hybrid

architectures to integrate the totality of their smart grid systems, as well as their emerging big data needs.

Further, massive data warehouses are difficult to support over the long term; often the best data architecture

designs are those that keep master data close to the processing engine/analytics, or vice versa.


The previously published GTM Research report The Smart Utility Enterprise concluded that only a hybrid architecture that achieves the following conditions will truly be equipped to implement a smart grid.

• Separates the data management; application logic and presentation into separate layers (i.e., is multitier)

• Has helper applications that surround legacy systems with new functionality

• Embraces service-oriented architecture (SOA) that encapsulates application functions into modular components for reuse

• Utilizes agent-based architecture for distributing intelligence to nodes like IED

• Supports big and unruly (i.e., unstructured) data

Finally, the following list of suggested best practices for utilities moving into smart grid was generated by systems integrator Accenture.



FiguRE 2-2: KEY BEST PRACTiCES FOR DEVELOPiNg AND iMPLEMENTiNg SMART gRiD SOLuTiONS

1. Recognize smart grid data classes and their characteristics to develop comprehensive smart grid data

management and governance capabilities.

2. Consider how data sources can support multiple outcomes via analytics and visualization to realize the

maximum value from the sensing infrastructure.

3. Consider distributed data, event processing and analytics architectures to help resolve latency, scale and

robustness challenges.

4. Consider the whole smart grid challenge when planning data management, analytics and visualization

capabilities—not just advanced metering infrastructure—to avoid stranded investments or capability

impediment.

5. Design data architectures that leverage quality master data to match data classes and analytics/

application characteristics. A giant data warehouse is rarely maintainable.

6. Look to new tools such as complex event processing to handle challenges around processing new data

classes. Managing the new smart grid data deluge via historical transaction processing approaches is likely

not scalable.

7. Develop business process transformation plans at the same time as—and in alignment with—smart grid designs.

SOuRCE: ACCENTuRE

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT Big Data And Analytics


3. BIG DATA AND ANALYTICS

3.1 Top 10 Smart Grid Drivers Of Big Data and Analytics

The following list identifies ten drivers that will likely increase the speed at which big-data and analytics technologies will be adopted in the utility industry.

FiguRE 3-1: TEN DRiVERS WHiCH WiLL MOVE uTiLiTES TO Big DATA AND ANALYTiCS

1. Utilities seeking ROI for advanced metering investments to justify the billions spent on AMI infrastructure.

2. The new technologies will improve the usefulness and granularity of demand-side management and

demand response programs in terms of better customer segmentation and other benefits.

3. The new technologies will improve asset management in an asset-intensive industry.

4. More data and analytics will lead to better grid operations management in extreme weather, including

reduced outage times, cost savings from better SAIFA and SAIDI indexes, and fewer dissatisfied customers.

5. The new technologies will lead to reduced energy theft and other non-technical losses.

6. The new technologies will smooth the integration of renewables and EVs.

7. The new technologies will facilitate the use of geospatial intelligence to visualize grid operations.

8. The new technologies will ease the strain being placed on traditional business intelligence (BI) and analytic

solutions from the exponential growth of data.

9. The speed of adoption will likely increase when key stakeholders in the utility industry acknowledge that

today’s utility enterprise IT architectures are not sufficient to meet future needs, specifically in terms of

their lack of cross-departmental data sharing capabilities.

10. New vendor technologies are driving shifts in terms of both what is affordable and what is possible.


THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT Vendor Profiles And Comparative Analysis


4. VENDOR PROFILES AND COMPARATIVE ANALYSIS

4.1 Introduction

In this section, we delve more deeply into the leading technology vendors offering solutions across the various subsectors and submarkets of the soft grid space, including data storage, data infrastructure, data management, and the growing application layer of smart grid.

2012 has presented utilities with a growing number of new offerings, including those with features such as geospatial visualization, cloud-based solutions, cluster analysis tools, intelligent alarm filtering capabilities, and others. These advances have generated a great deal of enthusiasm, as well as a considerable amount of confusion.

This seems like a fitting analogy for where the market stands today: it has thus far been successful in terms of creating a lot of excitement around big data and analytics, but hasn’t yet been equally successful in demonstrating either the capabilities or the business case for these new technologies. To some utility executives, the need for analytics is clearly obvious, but it appears that the majority of industry insiders in this traditionally change-averse industry still need a fair amount of education in order to be apprised of current and emerging technologies. Adding to the confusion is the fact that seemingly every company in the market – even those with little applicable experience in the field – is suddenly developing or offering ‘analytics’ products. Ultimately, however, both educational efforts and ROI prove-out will need to take place in order to spur investment and market proliferation.

However, after having spoken to dozens of utility executives on this topic, as well as conducting an extensive survey of more than 70 North American utility executives, it is clear that interest in these emerging technologies is now beginning to mount. Soon, it is likely that utilities will begin to adopt analytics technologies that will allow them to become more proactive in decision-making and to adjust their strategy based on the predictive views into the future that the technologies will facilitate. This will allow utilities to capitalize on the smart grid technologies that are now being deployed at scale; side-step potential problems; and better handle the steep challenges facing an industry in transition.



4.2 Vendor Taxomony and Vendor Rankings

FiguRE 4-1: LEADiNg VENDORS iN SOFT gRiD


4.2.1 Data Management and Movement Layer

The data management layer has been a focal point of this report, and along with the enterprise IT architecture that supports it, it represents both the biggest challenge and the biggest opportunity for today’s utilities. It is abundantly clear that we are now in the big data age, but how utilities will manage this paradigm shift remains to be seen. Across the industry, a gauntlet has been thrown down, and upstarts springing out of the distributed processing world of Hadoop see a multi-billion-dollar market up for grabs, as the need for real-time analytics in a world of massive, unstructured and complex data demands performance requirements above and beyond the capabilities of the legacy relational database management systems of yesteryear. Today’s data no longer fits neatly into columns and rows, and is likely to be generated on the terabyte- or petabyte-scale. As such, old and antiquated architectures are destined to fall.



Predictably, there are many vendors (and utilities) taking an “If it ain’t broke, don’t fix it” stance on the issue of data management. Year after year, companies like Oracle continue their incremental gains in the speed and performance of their relational database management systems. However, the emerging technologies are not ready to fully replace their predecessors. Principal Hadoop founder Doug Cutting describes his company’s platform as “augmenting and not replacing” regular databases. As should be expected, Oracle, Microsoft and others are experimenting with big-data products and platforms, but every database expert consulted for this report cautioned that at the moment, those offerings remain immature and experimental.

The implications of data management for smart grid are vast. Having said that, however, we don’t expect utilities to begin making large bets on technologies like Hadoop in the near term, for several reasons. First, the solution offerings are relatively young and utilities historically aren’t big risk-takers on new technologies. Second, and more importantly, utilities haven’t yet fully grasped the true value and potential of distributed data processing. This may be due to the fact that utilities’ first foray into dealing with big data – namely, smart meter data -- has relied upon meter data management systems that are based on older relational database management systems.

Over the past five years, utilities’ chief data concern has been ensuring that smart meter data flowed reliably into their CIS/billing systems, so that the utility could ensure payement. There have been some gestures and claims made in the industry about integrating siloed departments and building intelligent IT enterprises, but in truth, maintaining accurate and efficient billing standards has been the leading concern.

As 2013 approaches, meter data management systems have now been proven to be reliable, and many other concerns, including the question of the reliability of AMI networks, have largely been worked out. As a result, utility CIOs and data experts have been freed up to focus on extending smart grid into other applications. In undertaking this process, they will begin to re-examine how their data is architected and managed.



The immense wave of unstructured data that is coming to the grid in the near future is the real big data challenge. Up until now, the data that utilities have had to manage has been predictable. For example, utilities know when data from an average meter read will be sent, and roughly how big the resulting data will be. On the other hand, it is very difficult to anticipate the deluge of data that an extreme weather event will initiate, including inputs from DMS, OMS, integrated weather systems and other systems and sensors related to other grid assets that may be experiencing unusual performance. As the grid starts to send frequent status updates on all critical and non-critical assets, the only way to capture this data will likely be with advanced big-data tools.

As such, GTM Research believes that legacy RDBMS will be unable to meet the comprehensive future needs of the smart grid. Up until now, systems integrators and middleware players have been able to patch new solutions onto legacy systems, but at a certain point, big data will begin to overwhelm “spaghetti” architecture.

FiguRE 4-2: COMPARATiVE VENDOR RANKiNgS FOR THE DATA MANAgEMENT AND MOVEMENT LAYER

gTM RESEARCH

MAR

KET

BRE

ADTH

uTi

LiTY

RE

LATi

ON

SHiP

S

FuTu

RE N

EED

S

EXTE

NSi

BiL

iTY

FLEX

iBiL

iTY

SCAL

ABiL

iTY

REPO

RTi

Ng

&

ENTE

RPRi

SE T

OO

LS

CO

ST

SEC

uRi

TY

WEi

gH

TED

AVE

RAg

E

SAS 3 4 5 5 5 5 5 4 4 4.44

Teradata 4 3 5 5 5 5 5 4 4 4.44

IBM 5 5 4 4 4 4 5 3 5 4.33

EMC/Greenplum

5 5 4 4 4 3 5 3 5 4.22

Oracle 5 5 3 4 4 3 5 4 5 4.22

Cisco 5 5 4 4 3 3 4 4 5 4.11

SAP 5 5 4 4 4 3 5 3 4 4.11

Versant 3 3 5 4 5 5 4 4 4 4.11

Hortonworks 2 2 5 5 5 5 4 5 3 4.00

OSIsoft 3 3 4 4 4 5 4 4 5 4.00

Cloudera 2 2 5 5 5 5 3 5 3 3.89

Hadapt 2 2 5 5 5 5 3 5 3 3.89

5=highest score 1=lowest score



4.2.2 Analytics and Applications Layer

The analytics and applications layer covers the new and necessary solutions that vendors are bringing to the market. In the utility/smart grid space, there are four domains that will increasingly rely on analytics: the enterprise, grid operations (T&D), consumer-oriented offerings, and energy portfolio management and trading. The following diagram demonstrates where three of these domains sit relative to the physical grid infrastructure.

FiguRE 4-3: uTiLiTiES’ THREE PRiMARY DOMAiNS FOR ANALYTiCS

gTM RESEARCH

Virtually all smart grid vendors are competing in the analytics and applications layer. It is difficult to provide an apples-to-apples comparison of these vendors and their product and service offerings, as the solutions that each are offering are often unique. The following list identifies the leading solutions that vendors are targeting in the 2012-2015 timeframe.

FiguRE 4-4: LEADiNg uTiLiTY SMART gRiD ANALYTiCS FOR 2012 -2015

• Geospatial and visual analytics that offer a

centralized view of multiple technologies

• Outage restoration analytics

• Grid optimization and power quality (including

voltage control and conservation)

• Peak load management (via demand-side

management analytics) and energy portfolio

management analytics

• Asset protection analytics and predictive asset

maintenance

• Service quality analytics

• Vegetation management analytics

• Revenue protection (including theft and non-

technical loss analytics)

• Analytics to correct legacy system errors (such as

CIS and MDM)

• Consumer behavioral analytics (including

comparison to neighbors/peers)

• Home signature and thermostat control analytics

• Time-of-use pricing analytics

• Renewable energy and storage analytics


GENERATION

UTILITY INFRASTRUCTURE CONSUMER

TRANSMISSION SUBSTATION DISTRIBUTION HOME / BUILDING / DATA CENTERS

DISTRIBUTED GENERATION AND STORAGE

POWER LAYER INFRASTRUCTURE

COMMUNICATION LAYER END-TO-END COMMS PLATFORM

• Moving from Traditional, Historical Analytics

to Real-Time Predictive Analytics

• Complete Situational Awareness

• Business Intelligence (BI)

• Trading with “live look” at the Grid

Simulation/Visualization

ENTERPRISE ANALYTICS CONSUMER ANALYTICS

• Behavioral Analytics

• Tiered Pricing - Trading, Selling Megawatts (DR)

• Building Energy Management

• Power Analytics (Load Flow)

• Social Media Data Intergration

• DG/EV/Microgrid Analytics

GRID OPERATIONS ANALYTICS

Grid Optimization and Operational Intelligence

• Asset Management Analytics

• Crisis Management Analytics

• DMS Analytics

• Outage Management Analytics/Fault

Detection and Correction

• Weather/Location data

•Mobile Workforce Management

•Energy Theft



FiguRE 4-5: COMPARATiVE VENDOR RANKiNgS FOR DATA ANALYTiCS AND APPLiCATiON LAYER


MAR

KET

BRE

ADTH

uTi

LiTY

RE

LATi

ON

SHiP

S

iND

uST

RY-L

EAD

iNg

SO

LuTi

ON

VALu

E FO

R SE

RViC

ES/

SOLu

TiO

NS

FuTu

RE N

EED

S O

F C

uST

OM

ERS

EXTE

NSi

BiL

iTY

OF

FEAT

uRE

S

SOPH

iSTi

CAT

iON

OF

ANAL

YTiC

S

END

uSE

R EX

PERi

ENC

E

STR

ATEg

Y

WEi

gH

TED

AVE

RAg

E

SAS 4 4 5 5 5 5 5 4 4 4.60

IBM 5 5 5 4 4 4 4.5 4 5 4.53

Opower 3 5 5 5 5 4 4 4 5 4.45

Space-Time Insights 3 3 5 5 5 4 4.5 5 5 4.43

EcoFactor 3 3 4 5 5 5 4.5 5 5 4.38

GE 5 5 4.5 4 4 4 3 4 5 4.23

Siemens 5 5 4 4 4 4 4 4 4 4.20

eMeter (a Siemens co.) 3 5 5 4 4 4 3.5 4 5 4.18

Accenture 4 5 4.5 4 5 3 3 4 5 4.13

ABB/Ventyx 5 5 4.5 4 4 3 3.5 4 4 4.10

Landis+Gyr 4 5 4.5 4 4 4 3.5 4 4 4.10

Aclara 4 4.5 4.5 4 4 4 3.5 4 4 4.05

Tendril 3 4 4 4 4 4 5 4 4 4.05

Ecologic Analytics (a Landis+Gyr company)

3 4.5 4.5 4 4 4 3.5 4 5 4.05

Silver Spring Networks 4 5 4 4 4 4 3.5 4 4 4.03

Echelon 4 4 4 4 5 4 4 4 3 4.00

DataRaker 3 3 4 4 4 4 4 4 5 3.90

Telvent (a Schneider Electric co.)

4 5 4 4 3 4 3.5 4 3 3.83

EnerNOC 3 4 4 4 4 4 4 4 3 3.80

Itron 4 5 4.5 4 3 4 3 3 3 3.73

Tableau Software 3 2 4 4 4 4 4 4 4 3.70

Energate 3 3.5 4 4 4 4 3.5 4 3 3.68

Grid Net 3 3 3.5 4 4 4 3.5 4 4 3.65

Power Analytics 3 3 3.5 4 4 4 3 4 4 3.58

ECOtality 2 2 4 4 4 3 3.5 4 4 3.43

5=highest score 1=lowest score

THE SOFT GRID 2013-2020, SAS RESEARCH EXCERPT


FOR MORE iNFORMATiON ON Big DATA & uTiLiTY ANALYTiCS

Additional Resources

High-Performance Analytics for The Smart grid

This white paper presents results from a survey of more than 70 North American utility executives. The research reveals how utilities are defining, conceptualizing and understanding both big data and analytics. The paper also explores some of the barriers utilities face in both day-to-day use and enterprisewide adoption of analytics.

http://bit.ly/SASGTM

SAS Digital Magazine on Energy Transformation

This multi-media asset covers the hottest topics in today’s energy industry, including Dodd-Frank regulatory impacts, analytics for distribution/asset optimization, and exploration of unconventional oil and gas resources. To download the interactive magazine type this URL into the browser: http://bit.ly/energymag

Explore SAS Visual Analytics

SAS Visual Analytics provides unique insights that allow utilities to understand how customer and market behaviors influence drive profitable growth opportunities. With new online demos, a utility can experience how SAS Visual Analytics provides an in-depth knowledge of customers, assets and operations. Log on for a test drive today: http://bit.ly/utilityVA

Contact SAS

SAS is the leader in business analytics software and services. For over 35 years, our solutions have enabled utilities to find hidden patterns in data and create intelligence from disparate data sources for effective decision-making. Find out more at sas.com/utilities.

Tim FairchildDirector, SAS Global Energy [email protected] +1.919.531.0981

the soft grid 2013-2020: big data & utility analytics for ... · pdf filethe soft grid...

Documents