emc optimize infrastructure for ecologic analytics’ meter ... · the smart grid will lead to the...

White Paper

EMC Solutions Group

Abstract

This white paper highlights the value of the partnership between EMC and Ecologic Analytics. It includes technical reference architectures and test results for the EMC® optimized infrastructure supporting Ecologic AnalyticsTM Meter Data Management System (MDMS) for 10 million meters and two billion daily meter reads.

July 2012

EMC OPTIMIZED INFRASTRUCTURE FOR ECOLOGIC ANALYTICS’ METER DATA MANAGEMENT SYSTEM - TWO BILLION METER READS IN 11 HOURS EMC VNX7500 Unified Storage System

Achieved two billion meter reads in 11 hours for daily meter-to-cash processing

Best practices for unified storage configurations with FAST Suite

Ten million dual-channel meters providing IEC-CIM compliant reads

EMC Optimized Infrastructure for Ecologic Analytics’ Meter Data Management System Two Billion Meter Reads in 11 Hours

2

Copyright © 2012 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H10747

3 EMC Optimized Infrastructure for Ecologic Analytics’ Meter Data Management System Two Billion Meter Reads in 11 Hours

Table of contents

Executive summary ............................................................................................................... 5

Business case .................................................................................................................................. 5

Solution overview ............................................................................................................................ 5

Key results ....................................................................................................................................... 6

Introduction .......................................................................................................................... 7

Overview .......................................................................................................................................... 7

Purpose ........................................................................................................................................... 8

Goals ............................................................................................................................................... 9

Scope .............................................................................................................................................. 9

Audience ......................................................................................................................................... 9

EMC ................................................................................................................................................. 9

Ecologic Analytics .......................................................................................................................... 10

Terminology ................................................................................................................................... 10

Technology overview ........................................................................................................... 11

Introduction to components ........................................................................................................... 11

Ecologic MDMS application ............................................................................................................ 11

EMC VNX7500 storage system ....................................................................................................... 12

EMC FAST Suite .............................................................................................................................. 12

EMC FAST Cache ........................................................................................................................ 12

EMC FAST VP ............................................................................................................................. 13

EMC PowerPath .............................................................................................................................. 14

Configuration ...................................................................................................................... 15

Overview ........................................................................................................................................ 15

Physical environment ..................................................................................................................... 15

Hardware................................................................................................................................... 15

Solution software ...................................................................................................................... 16

Oracle configuration....................................................................................................................... 17

Linux kernel configuration .............................................................................................................. 18

MDMS ............................................................................................................................................ 19

Configuration ................................................................................................................................. 19

Testing and validation ......................................................................................................... 20

Testing methodology ...................................................................................................................... 20

Testing elements ............................................................................................................................ 20

Testing scenarios ........................................................................................................................... 21

FAST VP 2 Tier Baseline storage configuration ................................................................................ 21

FAST VP Two tier storage with FAST Cache ...................................................................................... 23


4

FAST VP configurations .................................................................................................................. 23

Test results ......................................................................................................................... 25

Overview ........................................................................................................................................ 25

Performance .................................................................................................................................. 25

Price-performance .......................................................................................................................... 26

Oracle AWR Report statistics .......................................................................................................... 27

System workload analysis .............................................................................................................. 28

Storage workload analysis ............................................................................................................. 29

Conclusion ......................................................................................................................... 31

Summary ....................................................................................................................................... 31

Findings ......................................................................................................................................... 31

References .......................................................................................................................... 32


Executive summary

Utility companies are preparing for an explosion in new data because of Advanced Metering Infrastructure (AMI) projects. Many utilities will be moving from monthly meter reads to hourly and even 15-minute intervals on multiple channels.

AMI can produce a staggering 6,000-fold increase in data. In addition, AMI is one of many new data-intensive applications. The smart grid will lead to the implementation of distribution automation, wide area measurement systems, demand response, and electric vehicles—all fueled by data.

Utilities are asking, “Where will we store all of this data and how will we manage it? How will we back it up? How will we protect it?” The challenge is not only one of data throughput, but more importantly, one of management. Utilities will need a new architecture.

EMC® and Ecologic Analytics TM together provide an integrated Meter Data Management System (MDMS) solution from application to database to backend infrastructure.

Ecologic Analytics provides a world class, scalable MDMS, which collects, validates, and manages thousands to billions of meter reads every single day. EMC provides the scalable, reliable, and high-performance infrastructure to store, protect, optimize, and secure the mission-critical and revenue-generating data assets.

This combined solution demonstrates an optimized architecture for 10,000,000 dual-channel meters providing 15-minute interval reads, processing two billion International Electrotechnical Commission-Common Information Model (IEC-CIM) 61968 Part 9 meter reads in 11 hours. EMC and Ecologic Analytics jointly tested and benchmarked this solution, which includes the following components.

The Ecologic MDMS:

Oracle 11g Release 2 Enterprise Edition Relational Database Management System (RDBMS) deployed in a three-node real application cluster (RAC) running on Cisco Unified Computing System (UCS) servers

Red Hat Enterprise Linux (version 5.5) operating system on the Oracle Database RAC nodes

Oracle Solaris x86 (version 10) operating system on the MDMS application processing server (virtualized using VMware)

EMC® VNX® 7500 Unified storage system, leveraging the following four configurations in separate tests:

EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP) with two storage tiers (SAS and NL-SAS )

FAST VP with three storage tiers (Flash, SAS, and NL-SAS)

FAST VP with two storage tiers (SAS and NL-SAS) and FAST Cache

FAST VP with three storage tiers (Flash, SAS, and NL-SAS) and FAST Cache

Business case

Solution overview


6

EMC tested the 10 million dual-channel meter load processing two billion daily meter reads with the Ecologic MDMS solution.

The tests revealed that EMC FAST Suite technologies, FAST Cache, and FAST VP with three storage tiers (Flash, SAS, and NL-SAS), independently resulted in unprecedented performance at scale with the best overall price-performance ratio. The tests combined FAST Suite components for optimizing the application thread processing of the Ecologic MDMS.

We achieved an interval meter read consumption of 82,687 per second and validation, estimation, and editing (VEE) processing of 199,588 meter reads per second.

Seven consecutive daily meter-to-cash processing days were executed, which yielded consistent results using just 11-hours of the 24-hour clock typically associated with meter-to-cash processing. This is significant because it allows ample headroom for processing unplanned events such as AMI network outages and operational anomalies.

Industry experience confirms that the test results reflect real-world production customer MDMS operations. The use of application threaded processing within the MDMS enables the Ecologic Analytics’ solution to scale down to mid-market and scale up to large-market requirements by configuring additional application processing threads. Production experience and test results indicate that Ecologic MDMS provides consistent performance that can be measurably improved using advanced storage technology, as shown Table 1.

Table 1. Ecologic MDMS processing times

MDMS process Meter reads per second

Register read consumption 12,356

Interval read consumption 82,687

Register and interval data VEE 199,588

Billing determinants creation 179,630

The total day’s end processing time was 11 hours.

Key results


Introduction

Utility companies are preparing for an explosion in new data because of Smart Metering or AMI. As utilities move from monthly meter reads to 15-minute intervals, the amount of data will increase by almost 6,000 times with only two channels of data! As you add more channels (kVARh delivered, kVAh delivered, voltage, current, and so on), the number grows by an additional factor.

Figure 1 shows the growth in meter reads because of AMI.

Figure 1. Storage requirements for a generic AMI single-channel meter deployment

This exponential increase in data requires a fundamentally new approach to meter data management.

The Electric Power Research Institute (EPRI) put forth that the goal of the smart grid is to enable multiple applications, but utilities must “first build the right foundation,”1 including security, network management, and data management.

Figure 2 shows EPRI's vision of the smart grid, including the three critical foundations in the middle.

1 Ibrahim, Erfan. Ph. D. “EPRI’s Smart Grid Vision & AMI/HAN Research Overview.” Electric Power Research Institute. Presented at EEI T&D Conference. April 7, 2009.

Overview


8

Figure 2. Electric Power Research Institute (EPRI) smart grid conceptual diagram

EMC and Ecologic Analytics have combined forces to provide the data management foundation as shown in the EPRI diagram in Figure 2. Customers are no longer asking for pieces of technology in silos; customers are asking their vendors to come together to offer complete solutions that include the MDMS, the storage configuration tuned for optimal performance, and the backup and recovery systems.

EMC and Ecologic Analytics worked together at EMC’s test facility to benchmark an optimized MDMS and storage infrastructure. The test case focused on a 10 million dual-channel, 15-minute interval meter reads implementation loaded with 13 months of daily data to represent a fully loaded, and steady state production system.

The purpose of this paper is to demonstrate our ability to scale a solution to support very large utilities and to determine the optimal price-performance storage configuration for the Ecologic MDMS.

In September 2011, EMC and Ecologic tested an optimized infrastructure for 500,000 dual-channel meters providing 15-minute interval reads to provide a reference architecture for mid-sized utilities2.

EMC and Ecologic conducted this new benchmark to demonstrate market-leading performance at unprecedented scale. The two benchmarks we conducted demonstrated that the technical infrastructure and generally available Ecologic MDMS software solution provides elastic scalability by addressing the needs of both a mid-sized utilities and very large-sized utilities. Whether the utility has 500,000 or 10,000,000 meters, our system helps it to complete its daily processing within the allotted windows. This ensures successful meter-to-cash operations, even when new huge volumes of data are added as a result of AMI and Smart Grid. This infrastructure allows utilities to focus more of their time and effort on higher-level projects like

2 You can find that report on EMC’s online support website, entitled: EMC Optimized Infrastructure for Ecologic Analytics’ Meter Data Management System

Purpose


analytics when they feel confident that this infrastructure is helping their production processes run smoothly.

The MDMS contains meter data from 10 million dual-channel meters all providing a daily register read and 15-minute interval reads. To emulate a real-world scenario, two percent of the reads were missing and estimated, and one-half of one percent of the reads were “bad” (suspected as being inaccurate) and estimated. The application deployment and technology configuration represents existing customers in production. The MDMS used 96 application-processing threads, which is optimal for a large-scale utility represented by this benchmark and the technical infrastructure deployed.

EMC and Ecologic Analytics sought to improve MDMS application per-thread performance for core meter-to-cash MDMS processing using advanced storage technology. The core functions include:

Data synchronization with Customer Information System (CIS) processing

Register and interval meter data consumption

Validating, estimating, and editing (VEE) processing

Billing determinant creation

This solution includes building an EcoLogic MDMS environment on EMC systems to demonstrate optimal configurations for storage and data protection.

This solution does not provide a comprehensive guide to every aspect of an Ecologic solution nor does it seek to demonstrate the maximum performance of an EMC storage system for all workloads. This white paper provides a reference architecture for achieving optimal performance on the storage system for the particular Ecologic MDMS workload. This paper is for anyone engaged in or evaluating advanced metering infrastructure or meter data management (MDM) system projects. It is important for IT personnel to right size the IT systems necessary to support these mission-critical applications and utility processing objectives. It is also important for the business to understand the IT implications of such projects.

EMC Corporation is the world's leading developer and provider of information infrastructure technology and solutions that enable organizations of all sizes to transform the way they compete and create value from their information. EMC has served investor-owned utilities, municipalities, and cooperatives for over 25 years with solutions that address their information management challenges. You can find more information about EMC products and services at www.EMC.com/smartgrid.

Goals

Scope

Audience

EMC

http://www.emc.com/smartgrid


10

Ecologic Analytics, LLC is the leading provider of meter data management systems based on international standards, serving electric, natural gas, and water utilities. Today, the Ecologic MDMS validates hundreds of millions of meter reads every day for leading Smart Grid utilities, transforming the data consumed from AMI/smart meter endpoints into accurate, timely, and actionable information for decision making across the utility. The company, founded in 2000 and now a division of Landis+Gyr, has its base of operations in Bloomington, Minnesota. For more information, visit www.EcologicAnalytics.com.

This paper includes the following terminology.

Table 2. Acronyms

Term Definition

AMS Asset Management System

AWR Automatic Workload Repository

CIS Customer Information System

FAST VP Fully Automated Storage Tiering Virtual Pools

GIS Geographic Information System

LP Load profile

MDMS Meter data management system

NL-SAS Near-line-SAS drives

OMS Outage Management System

SAN Storage area network

SCADA Supervisory Control and Data Acquisition

VEE Validating, estimating, and editing

WMS Workforce Management System

Ecologic Analytics

Terminology

http://www.ecologicanalytics.com/


Technology overview

This section details the components that make up the solution described in this white paper, which includes:

Ecologic MDMS 2.8 application

EMC VNX7500 storage system

EMC FAST™ Suite

EMC PowerPath® V5.6

Cisco UCS C460-M2

The Ecologic MDMS consolidates many data streams into a manageable information flow, allowing all stakeholders secure access to the wealth of validated and stored AMI data offered in formats usable by other enterprise solutions.

We deployed the Ecologic MDMS technical architecture using the following software components:

Oracle 11g Release 2 Enterprise Edition RDBMS

Oracle 11g Release 2 Enterprise Edition Grid Infrastructure

Red Hat Enterprise Linux (version 5.5) on Oracle RAC servers

Oracle Solaris x86 (version 10) operating system on the MDMS application processing server

Introduction to components

Ecologic MDMS application


12

The MDMS application is designed to use three tiers as represented in Figure 3 below.

Figure 3. Ecologic MDMS architecture

The EMC VNX7500 storage system combines five 9s (99.999%) availability with innovative technologies like FAST, Flash drives, Virtual Provisioning™, a 64-bit operating system, and multi-core processors. The VNX7500 scales up to 2,970 terabytes (TB) of storage to handle future growth and features UltraFlex™ technology, providing multiple protocol options and online-expandable connectivity.

The VNX series has been expressly designed to take advantage of the latest innovation in Flash drive technology. The combination of Flash drives and hard disks delivers improved performance and efficiency while minimizing cost per GB. The EMC FAST Suite contains the software necessary, specifically FAST Cache and FAST VP, to improve performance and maximize storage efficiency on the VNX. With only a few Flash drives and the FAST Suite, EMC VNX customers can benefit from a FLASH 1st data strategy, ensuring that that highly active data is stored on and served from Flash drives, either in the cache or in the pool, for optimal application performance. while less active data is tiered on HDD’s for the lowest TCO. See Figure 4.

EMC FAST Cache A caching tier is a large-capacity secondary cache that uses Flash drives positioned between the storage processor's dynamic random-access memory (DRAM)-based primary cache and hard-disk drives. EMC FAST Cache is a non-disruptive, read/write cache that extends the VNX’s existing cache by up to 2 TB. FAST Cache monitors incoming I/O for access frequency and automatically copies frequently accessed data in 64k chunks from the back-end drives into the cache. FAST Cache is easy to

EMC VNX7500 storage system

EMC FAST Suite


administer and cost-effectively provides immediate performance benefits to the system.

EMC FAST VP FAST VP manages the dynamic tiering of data across a storage pool made up of more than one drive type for optimal disk utilization and efficiency. Based on customer-defined policies, FAST VP’s software algorithmically promotes and demotes user data within the pool, based on how frequently it is accessed. More frequently accessed data is tiered to higher performance drives such as Flash. Infrequently accessed data moves to low-cost, high-capacity tiers such as SAS or NL-SAS drives. Over time, the most frequently accessed data resides on the fastest storage devices, and infrequently accessed data resides on economical bulk storage. FAST VP provides both capital expenditures (CAPEX) and operating expenditures (OPEX) benefits to customers by allowing them to purchase a mixed drive allocation, which results in lower power and cooling costs, a smaller data footprint, as well as decreased administration time.

Figure 4. EMC’s FLASH strategy leverages the FAST Suite

For more details on FAST VP and FAST Cache, contact your EMC representative or refer to the following white papers on the EMC online support website:

EMC FAST Cache—A Detailed Review

EMC FAST VP for Unified Storage Systems—A Detailed Review

IDC: Quantifying the Business Benefits of EMC VNX FLASH 1st


14

EMC PowerPath automates path failover and recovery and optimizes load balancing so that you can get the most out of your data center environment. It automatically tunes your storage area network (SAN) and selects alternate paths for your data, if necessary. Residing on the server, PowerPath Multipathing enhances SAN performance and application availability.

EMC PowerPath


Configuration

This solution includes a three-node Oracle RAC cluster running the Oracle database. The Ecologic MDMS processing server is hosted on a VMware virtual machine running Solaris 10 x86. The Oracle servers connect to the SAN through redundant Fibre Channel (FC) host bus adapters for multipath I/O for high performance and availability. The Oracle cluster interconnect is over a 10 Gb private network that we used for NFS access to the VNX during the load process.

The back-end storage is an EMC VNX7500 array connected to the SAN switches.

Figure 5 displays the physical hardware architecture in this solution.

3

Figure 5. Physical environment overview

Hardware Table 3 lists the hardware used in this solution. 3An Oracle best practice recommends a private 10 Gb network for the Oracle interconnect. For the purposes of this

solution, we used the 10 Gb network for the load process when the cluster was inactive.

Overview

Physical environment


16

Table 3. Solution hardware

Hardware Quantity Configuration

Storage array 1 EMC VNX7500 running OE 05.31.000.5.509

For details on storage configuration, see the section entitled “Testing scenarios” on page 21.

Network n/a SAN: 8 Gb FC redundant director switches

High speed LAN: 10 Gb Ethernet switch

Access LAN: 1 Gb Ethernet switch

Database servers 3 Rack mount server

Four 8-core Intel® Xeon® CPU E7 4830 @ 2.13 GHz (32 cores per server)

128 GB physical memory

2 Dual-Port QLogic QLE2562 8Gb FC HBAs

Application processing server (VMware ESXi server)

1 Intel® Xeon® CPU X5570 @ 2.93 GHz

Virtual machine with 4 virtual CPUs 2 GB of memory

Solution software Table 4 describes the software used in this solution.

Table 4. Solution software

Software Version

Ecologic Analytics MDMS 2.8

Oracle Database 11g R2 Enterprise Edition with Partitioning option for Linux x86_64

11.2.0.2.0 – 64-bit production

EMC PowerPath 5.6

Oracle Grid Infrastructure 11g R2 Enterprise Edition including ASM and Clusterware for Linux x86_64

11.2.0.2.0 – 64-bit production

Oracle ASMLib 2.0

Red Hat Enterprise Linux 5.5 64-bit

Solaris 10 (x86)


Table 5 shows the Oracle configuration settings used in this solution.

Table 5. Oracle ASM instance initialization parameters with non-default values

Parameter Setting

large_pool_size 12 M

memory_max_target 272 M

memory_target 272 M

processes 100

sessions 192

sga_max_size 272 M

sga_target 0

shared_pool_reserved_size 7969177

shared_pool_size 0

sort_area_size 65536

use_large_pages TRUE

workarea_size_policy AUTO

Table 6 shows the Oracle ASM disk groups configurations used in the solution

Table 6. Oracle ASM disk group configurations

Disk group name ASM redundancy

AU size

Files

+DATA external 16 M Datafiles, temp files

+OLREDO11 external 1 M Online redo log files of node 1






+VOTE high 1 M Voting disk files and CRS files

+ARCH external 1 M Archived redo log files

Oracle configuration


18

Table 7 shows the Oracle configuration settings.

Table 7. Oracle configuration settings

Configuration Setting

Oracle Flashback Off

Oracle Archive log On

Redo logs Two ASM disk groups per node, 10*16 GB online redo log files per ASM disk group

Total database size 200 TB

Oracle version 11.2.0.2.0 – 64-bit production

Table 8 shows the Oracle database initialization parameters and settings.

Table 8. Oracle database instance initialization parameters with non-default value

Parameter Settings

db_block_size 16384

db_file_multiblock_read_count 256

filesystemio_options SETALL

job_queue_processes 1000

log_archive_max_processes 2

log_buffer 268419072

open_cursors 300

parallel_min_servers 96

pga_aggregate_target 52 GB

processes 1000

session_cached_cursors 100

sga_target 64 GB

undo_management AUTO

Table 9 shows the Linux hugepages kernel parameter setting.

Table 9. Linux hugepages kernel parameter setting

Configuration Setting

vm.nr_hugepages 33024

Linux kernel configuration


During the tests, the application ran on the application processing virtual machine and the database ran on the Oracle RAC cluster.

Table 10 describes the key MDMS processing operations.

Table 10. MDMS meter-to-cash processing operations

Function Description

Master data synchronization Provisioning meters, accounts, locations, and routes with inbound interfaces from CIS, outbound to AMI

Register data consumption Consumption of register (anchor) reads from AMI

Interval data consumption Consumption of interval load profile (LP) reads from AMI

Register data VEE Validation, estimation, and editing register data

Interval data VEE Validation, estimation, and editing interval data

Billing determinant creation Outbound interface to CIS containing billing determinant data

Table 11 describes the metrics used in the MDMS environment.

Table 11. MDMS Environment

Item Metric

Number of AMI meters 10,000,000

Number of channels per AMI meter 2

Number of historical months of daily meter reads loaded before measuring performance tests

13

Daily register reads 20,000,000

Daily interval reads 1,920,000,000

Total daily reads 1,940,000,000

Daily percentage of missing reads which were estimated

2%

Daily percentage of bad reads which were estimated

0.5%

Total daily reads estimated by VEE processing

48,500,000

Daily billing determinants reads processed (1/20th of the meter population per billing cycle)

97,000,000

MDMS Configuration


20

Item Metric

Daily AMI meters and service delivery points added during performance testing (simulates new meters deployed in the field)

10,000

Testing and validation

EMC ran the same daily processing that a utility would carry out as described in Table 10 on page 19.

We gauged application performance by the total duration of the operations; a shorter duration indicates better performance. We collected the performance statistics from the following sources:

Log files identifying the start and stop times of each operation.

Automatic Workload Repository (AWR) reports

Performance statistics from the storage system

We correlated these statistics with the operation execution times and performed an analysis of I/O characteristics. This resulted in a great deal of insight into the application’s I/O profile.

We executed these operations for seven consecutive daily processing cycles to demonstrate that application and system performance remained consistent and did not degrade.

Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Do not use this workload as a substitute for a specific customer application benchmark when contemplating critical capacity planning and/or product evaluation decisions.

We obtained all performance data contained in this report in a rigorously controlled environment. Results obtained in other operating environments may vary significantly.

EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.

EMC and Ecologic Analytics ensured that the testing environment was as similar as possible to a realistic production environment by:

Turning on Oracle Archive Log

Using a three-node Oracle RAC cluster (fault tolerant, scalable)

Using 96 MDMS application processing threads spread across the cluster

Testing methodology

Testing elements

Notes


Storing all data on the same storage platform (rather than placing datafiles on one array and redo logs on another)

Processing two percent missing reads for both register and interval data; one-half of one percent “bad” reads for both register and interval data (all estimated)

In this solution, the EMC solutions team tested four different storage configurations:

FAST VP 2 Tier Baseline storage configuration: The baseline storage configuration consisted of FAST VP with two storage tiers, NL-SAS and SAS.

FAST VP 2 Tier and FAST Cache: the second configuration adds FAST Cache in addition to the FAST VP two-tier baseline.

FAST VP 3 Tier: The third storage configuration leverages FAST VP with three storage tiers, NL-SAS, SAS, and FLASH

FAST VP 3 Tiers and FAST Cache: The fourth configuration adds FAST Cache in addition to FAST VP thee tier storage.

Figure 6 shows the baseline storage configuration for the tests performed. As the name suggests, this configuration is the comparison point for the three subsequent system designs.

Testing scenarios

FAST VP 2 Tier Baseline storage configuration


22

Figure 6. FAST VP two-tier baseline storage configuration

Each volume is composed of two or more disks, depending on the protection scheme and purpose of the volume. Database volumes contain the database files, which we placed on RAID 5 and optimized for read performance for the given capacity. According to EMC best practices, RAID 5 has excellent random read performance and has the best ratio of available capacity for parity-protected RAID groups. We built the


Oracle archive volumes for high capacity using NL-SAS disks that we configured for the highest write performance by leveraging RAID 1/0.

We built the Oracle Redo log volumes for the highest possible write performance by using SAS disks leveraging RAID 1/0 in an 8+8 configuration.

EMC does not recommend placing Oracle datafiles and log files in the same disk group for the following reasons:

Reliability: The logs play a pivotal role in Oracle database recovery. If datafile corruption occurs, the database administrator can go back to an older copy of the datafile and apply the logs. Similarly, if logs are lost, the Oracle database can guarantee zero or minimal data loss if online redo logs are located on a different set of spindles.

I/O type and size: The I/O profile of log files tends to be sequential. By mixing log files with datafiles, the sustained write bandwidth of a drive drops as the spindle begins to “seek” more often.

Performance: Redo log writes are synchronous and are required to complete in the least amount of time. By placing them on separate storage devices, the commit writes do not have to share the LUN I/O queue with large asynchronous buffer cache checkpoint I/Os. Having the logs on their own devices makes it possible to use one RAID protection type for datafiles (RAID 5) and another for logs (RAID 1/0).

The second configuration adds FAST Cache on top of the FAST VP two-tier baseline. It uses the same configuration as the baseline with the addition of sixteen Flash drives, serving as a second-level cache. The team used EMC internal performance analysis tools to determine the size of the FAST Cache.

The benefits of FAST Cache occur with the random read and write profile of the database files. The I/O patterns for Oracle logs, both online and archive tend to be sequential in nature. The write cache of the storage system can coalesce these small writes into bigger back-end I/O stripes, allowing the more cost-effective spinning hard disks to handle that load, and freeing the higher-performing flash drives for large database files.

You must provision Flash drives for FAST Cache in pairs and assign RAID 1 protection according to EMC best practices.4 FAST Cache uses RAID 1 (mirroring) to provide both read and write caching.

Figure 7 shows the meter data management disk pool layouts we used for VNX FAST VP and FAST Cache testing. All three configurations with FAST VP use two disk pools, one for the database and one for the archive data. A pool is a collection of disks that can share the storage load, and the distribution of that load depends on the nature of the data. As in this scenario, multiple disk technologies, or tiers (Flash, SAS, and NL-SAS) can exist in the same pool.

4 “EMC Unified Storage Best Practices for Performance and Availability Common Platform and Block Storage 31.0.”

Applied Best Practices. Revised: 06/23/2011. Page 45-46.

FAST VP Two tier storage with FAST Cache

FAST VP configurations


24

We created the redo, cluster registry, and Ecologic MDMS application volumes on VNX RAID groups. These diagrams display the disk types used and RAID protection from the perspective of the database file systems. During FAST-VP two-tier with FAST Cache testing, we assigned 1600 GB of additional FAST Cache to the database pool (16 200 GB, RAID 1-protected disks). During FAST-VP two-tier with FAST Cache testing, we assigned 733 GB of additional Flash Cache to the database pool (16 200 GB, RAID 1-protected disks).

Figure 7. Storage layout

Refer to Table 6 on page 17 for a description of how we configured the ASM disk groups on the volumes.


Test results

All three FAST configurations achieved the desired results, with each providing daily processing times in less than 12 hours. Specifically, FAST three-tier with FAST Cache provided the most significant performance improvement relative to the baseline configuration. However, from a performance relative to price consideration, the FAST two-tier with FAST Cache showed the best performance return on a customer’s investment. All three FAST configurations can support large-scale customers running the Ecologic MDMS. This section describes the results and provides context behind the recommendations in this paper.

There are multiples ways to interpret the testing results of this solution. This section includes the results and describes the various ways to view them, including by:

Performance

Price-performance

Figure 8 shows the performance of each configuration we tested. A shorter duration yielded better performance. The three FAST Suite configurations all show improved performance over the FAST VP two-tier baseline. The FAST VP three-tier with FAST Cache configuration yielded the best performance. It was 33 percent better than the FAST VP two-tier baseline.

Figure 8. Performance results

Overview

Performance


26

Often when performance numbers are stated, the next question that comes up is price. It is usually not too difficult to design a system that performs well if money is no object, but that is rarely the case. We typically use “price-performance,” or performance relative to price, as a measurement of the solution’s effectiveness5.

A shorter duration in runtime is better, and of course, a lower price is better, therefore we can multiply these two values together, with the lowest result being the best price-performance choice.

Figure 9 shows that the FAST VP two-tier configuration with FAST Cache provides the overall best price-performance at 16 percent better than the baseline. The baseline includes FAST VP, two-tiered storage with no Flash drives. We scaled the various configurations relative to the baseline to show their price-performance.

Figure 9. Normalized price-performance

5 This includes storage hardware and storage software components in the price (VNX storage, fast suite licenses,

pricing per disk, etc.)

Price-performance


The results discussed thus far focus on performance (measured by job duration) and the price of the various configurations. Wait time is another performance metric often used when evaluating database applications. Table 12 shows the database wait times for the longest running, most compute, and I/O intensive job (interval meter data consumption) run during a processing day. We retrieved these database statistics from Oracle AWR reports for a single processing day.

Table 12 shows how enabling FAST Cache greatly reduces wait times for read activity. When comparing the database file sequential read events between the baseline and FAST Cache runs, the wait time drops from 36.27 ms to 21.41 ms, an increase in performance of 47 percent. By reducing the individual wait times, we also reduce the total wait time from 484,972.62 ms to 141,691.00 s. For database parallel write events, it dropped from 7.08 ms to 5.31 ms, with an increase of 25 percent after enabling FAST Cache. The total wait time dropped from 151,930.73 ms to 89,855.85 s.

Table 12 also shows how the FAST VP three-tier configuration improves the I/O performance through a different method than FAST Cache. FAST VP successfully identified the working set (hot data) based on the array I/O statistical analysis. FAST VP then promotes the hot blocks to a higher performance tier of storage, FLASH, or SAS. FAST VP also demotes any blocks of data that are already in the higher performance tiers if they are no longer in high demand.

By combining FAST Cache and FAST VP, we achieved maximum performance by using the best characteristics of each product.

Table 12. Top database wait events

Event

FAST VP 2 Tier (baseline)

FAST VP 2 Tier plus FAST Cache

FAST VP 3 Tier FAST VP 3 Tier plus

FAST Cache

Avg. wait time/ms

Total wait

time/s

Avg. wait time/ms

Total wait

time/s

Avg. wait time/ms

Total wait

time/s

Avg. wait time/ms

Total wait

time/s

Db file sequential

read 36.27 484,972 21.41 141,691 10.01 68,862 10 68,284

Db file scattered

read 25.79 87,583 20.91 52,254 12.18 31,286 12 31,234

Db file parallel

read 70.37 25,163 27.72 9,187 10.33 3,298 38 137,591

Direct path read

temp 5.44 15,893 4.06 12,138 2.04 5,888 3 * 9,075 *

Db file parallel write

7.08 151,931 5.31 89,856 4.68 69,279 3.25 * 52,414 *

No longer in the top wait events

Oracle AWR Report statistics


28

Figure 10 combines multiple statistics for one day’s processing of the FAST VP three-tier configuration into a single view. The bottom chart depicts durations of the MDMS processing operations in order of execution over time. The middle chart shows the maximum 5-minute load average total over the three cluster nodes. The top chart shows the maximum 5-minute averages of read and write IOPS for each MDMS processing operation.

The interval read consumption-operation ran for the longest duration, drove the highest write IOPS, and had the highest CPU load averages. The interval VEE operations drove the highest read IOPS.

We configured this solution for both performance and resiliency. The peak load average across the cluster indicates that there is ample headroom for the workload to complete in an acceptable period if one of the three cluster nodes fails. We validated this by executing a full day of meter-to-cash daily processing while one of the Oracle RAC nodes failed and was unavailable, which completed in 14 hours.

Figure 10. MDMS operations and resource utilization

System workload analysis


Over the course of the day, the MDMS processing operations impose a varying workload on the storage system. Figure 11 shows the total read and write I/O generated by the Oracle RAC hosts during the FAST VP three-tier testing. At the tail end of the interval read consumption operation, we see a sustained write workload. The interval VEE processing operations during the last few hours exhibit a sustained read workload.

Figure 11. Read and write IOPS (host LUNs)

Storage workload analysis


30

The database files were stored on a three-tier FAST VP storage pool. Figure 12, by design, shows that the combination of the Flash and SAS drives provide nearly all the IOPS. The NL-SAS provides the high capacity for the infrequently accessed data within the 200TB database.

Figure 12. Total IOPS by disk drive type (back-end disks)


Conclusion

The team drew the following conclusions from the solution testing:

When built on EMC storage systems with FAST technology, the Ecologic MDMS application can consume 12,356 register and 82,687 interval IEC-CIM compliant meter reads per second and perform VEE processing at 199,588 meter reads (register and interval) per second. EMC and Ecologic Analytics provide a cost-effective platform for large-market utilities deploying smart meters.

Interval data consumption and VEE processing consume most of the processing activity within the MDMS application, so improving the performance and cost efficiency of these operations to provide the maximum return on investment.

Using Ecologic MDMS application-threaded processing with EMC FAST VP as the storage configuration provides the best performance for the cost.

While all the tested FAST and FAST Cache configurations improve performance over the FAST VP two-tier baseline, the configuration with FAST VP two-tier configuration with FAST Cache provides the best price-performance. This configuration improves performance by 24 percent over the FAST VP two-tier baseline configuration and, simultaneously, has a 16 percent better price-performance.

When performance is the top priority, FAST VP with three storage tiers and FAST Cache is the best choice.

Both FAST VP and FAST Cache help the Ecologic MDMS because a reasonably priced Flash storage tier can service some of the “working set,” or data responsible for a majority of the I/O. A great benefit of the advanced FAST VP and FAST Cache technologies is that the storage system automatically places the most active data on the highest storage tier, while placing the least frequently accessed data on the lowest tier.

Industry experience confirms that the test results reflect real-world production customer MDMS operations. The use of application threaded processing within the MDMS enables the Ecologic Analytics’ solution to scale down to small and mid-market utility companies and scale up to large-market utilities by configuring additional application processing threads. Production experience and test results indicate that Ecologic MDMS provides consistent performance that can be measurably improved using advanced storage technology, as shown in Table 1 on page6.

Summary

Findings


32

References

For more information, see the following white papers (some are available on the EMC online support website). If you do not have the required access, contact your EMC representative.

EMC Optimized Infrastructure for Ecologic Analytics’ Meter Data Management System

EMC Unified Storage Best Practices for Performance and Availability: Common Platform and Block O.E. Storage 31.5 - Applied Best Practices

EMC FAST Cache A Detailed Review

IDC: Quantifying the Business Benefits of EMC VNX FLASH 1st

emc optimize infrastructure for ecologic analytics’ meter ... · the smart grid will lead to the...

Documents