accelerating oracle oltp 12c database performance with...

Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN This white paper describes how the performance of an Oracle 12c Online

Transaction Processing (OLTP) database can be accelerated using

Dell Fluid Cache for SAN.

Technical White Paper

Krishna Kamal Kapa

Dell Database Solutions Engineering

April 2014

2 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND

TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2014 All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of

Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the Dell logo, PowerEdge, PowerVault, OpenManage, EqualLogic, Compellent, KACE, FlexAddress, are trademarks of Dell

Inc. Other Dell trademarks may be used in this document. Intel, Pentium, Xeon, Core and Celeron are registered trademarks of

Intel Corporation in the U.S. and other countries. Microsoft, Windows, Windows Server, Internet Explorer, MS-DOS, Windows

Vista and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or

other countries. Red Hat and Red Hat Enterprise Linux are registered trademarks of Red Hat, Inc. in the United States and/or

other countries. Novell and SUSE are registered trademarks of Novell Inc. in the United States and other countries. Oracle is a

registered trademark of Oracle Corporation and/or its affiliates. Citrix, Xen, XenServer and XenMotion are either registered

trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries. Broadcom and NetXtreme are

registered trademarks of Broadcom Corporation. Qlogic is a registered trademark of QLogic Corporation. Other trademarks and

trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and

are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others.

April 2014| Version 1.0


Table of contents

Executive summary ................................................................................................................................................... 4

Introduction ............................................................................................................................................................... 4

Dell Fluid Cache for SAN overview ........................................................................................................................ 5

Solution design and reference architecture ......................................................................................................... 6

Traditional rotational hard drive-based storage solution .................................................................... 6

Fluid Cache for SAN solution ..................................................................................................................... 7

Test methodology ..................................................................................................................................................... 8

Performance results and analysis ........................................................................................................................... 9

Conclusion ................................................................................................................................................................ 11

Acknowledgement ...................................................................................................................................................12

References .................................................................................................................................................................12

Improving Oracle OLTP database performance with Dell Fluid Cache for DAS ............................12

Hardware configuration details .............................................................................................................................13

Server configuration ..................................................................................................................................13

Storage configuration ............................................................................................................................... 15

Network Switching Hardware .................................................................................................................. 15

Software configuration details ............................................................................................................................... 15

Oracle configuration settings ................................................................................................................. 16

Kernel parameter settings ........................................................................................................... 16

User Security Limits Settings ....................................................................................................... 17

Database parameter settings ....................................................................................................... 17

Udev settings Settings for Baseline Configuration .................................................................. 17

Udev Settings for Fluid Cache Configuration.......................................................................... 18

Server Configuration Profile .................................................................................................................... 18


Executive summary

Application performance may be limited by multiple factors such as low interconnect speeds, slow

drive response times, cumbersome software stacks, and maxed out storage controllers. Application

performance acceleration is key to business and customer success, and can be achieved in multiple

ways. Adopting SSD technology and server caching software in your data center are a few ways to

improve application performance. Dell has built an enterprise-class caching solution named “Dell Fluid

Cache for SAN” that can radically accelerate application performance.

Dell Fluid Cache for SAN is a highly available write-back server-side caching accelerator that utilizes

high speed PCIe SSDs as a shared, caching solution with an “open” compute cache pool layer. Fluid

Cache for SAN runs in a flexible and low latency Dell Compellent™ SAN environment. The key

differentiations of Fluid Cache for SAN from other host-side caching solutions are its global cache

pool distributed across multiple servers, write back cache policies, easy access media and fully

integrated with the storage array features and management.

This white paper can help you configure the Dell Fluid Cache for SAN solution and demonstrates how

the performance of Oracle®

Database Online Transaction Processing (OLTP) can be accelerated using

this solution compared to the traditional hard disk-based solution.

Figure 1: Advantages of Fluid Cache for SAN in a 3 node Dell lab test of a hardware stack enabled with Dell Fluid Cache for SAN on an OLTP cluster running on an Oracle database

Introduction

This white paper studies the performance of an Oracle OLTP database in a Fluid Cache for SAN

solution. It compares the performance of the Fluid Cache solution against a traditional storage

solution (without any host based caching).

The key benefits of Fluid Cache for SAN compared to the traditional solution are:

Increased transactions per second (TPS)

Decreased average response times (ART)

On-the-fly scaling to meet growing business demand

No single point of failure from server to SAN.

4.4x more transactions per second

9.6x reduction in average response time at one sec SLA

~4x more concurrent users at one second or less average response time


Dell Fluid Cache for SAN overview

Dell Fluid Cache for SAN is server-side caching accelerator software. Dell Fluid Cache for SAN makes

high-speed PCIe SSDs a shared, distributed cache resource. Dell Fluid Cache for SAN is deployed on

clusters of Dell PowerEdge™ systems within a SAN environment and connected using RoCE-enabled

Ethernet adapters.

A minimum of three compatible Dell PowerEdge servers systems supporting Dell PCI Express Flash

SSDs are required to configure a Fluid Cache cluster. Dell validates up to eight systems per cache pool

with a maximum of 1.6TB of cache per server, and a maximum capacity of 12.8TB of cache per cache

pool.

Figure 2: Fluid Cache for SAN Components

Dell Fluid Cache for SAN uses three types of networks:

Cache Network: A high speed, low latency private network used by Fluid Cache to

connect the PowerEdge systems in the Fluid Cache cluster.

Management Network: The connection to Dell Compellent Enterprise Manager,

which manages Fluid Cache and the SAN.

SAN Network: The network that Dell Compellent Storage Center uses to handle data

connectivity within the SAN.


Solution design and reference architecture

This section provides the solution design of Fluid Cache for SAN and traditional rotational hard drive-

based storage solution used as the baseline configuration.

Traditional rotational hard drive-based storage solution

The baseline configuration is deployed on two PowerEdge R820 systems connected to Compellent

SC8000 controller and SC220 expansion storage using Fibre Channel Brocade 6510 16Gbps switches.

Redundant Dell Networking S4810 10GbE switches are used for Oracle RAC Cache Fusion. The SC220

enclosure is populated with 20 x 300GB 15k RPM SAS drives as Tier 1 and 4 x 1TB SATA 7.2 RPM drivers

as Tier 2.

The two systems are installed with Red Hat®

Enterprise Linux®

6 Update 4 and Oracle Cluster Database

12c with a pluggable database deployed on the two nodes. The DATA diskgroup containing database

files, controlfiles, and redolog files are placed in Tier 1 storage and FRA diskgroup containing backups

and archive logs are placed in Tier 2.

Figure 3: Baseline Configuration Architecture


Fluid Cache for SAN solution

Dell Fluid Cache for SAN cluster is deployed with Dell Fluid Cache software on two PowerEdge R820

systems and one PowerEdge R620 system which is added as a management server for Fluid Cache.

The two PowerEdge R820 systems act as cache servers by hosting 2 x 350GB Express Flash PCIe SSDs

in each server.

The Fluid cache clustered servers are connected to Dell Networking S4810 10 GbE low latency

network using Mellanox®

ConnectX®

card for the cache network of Fluid Cache. Using this network,

the cached blocks are transferred across the Fluid Cache cluster.

Dell Compellent Enterprise Manager monitors, manages and analyzes storage center and Fluid Cache

clusters from a centralized management console. Compellent Enterprise Manager is installed on a

dedicated Windows system for monitoring and enabling the cache to the Compellent disks.

Fluid cache for SAN utilizes Dell Express Flash PCIe SSDs and Dell Compellent storage and moves the

most frequently accessed data closer to compute resources minimizing data travel from storage

through the network, improving response time and accelerating both reads and write performance.

Figure 4: Fluid Cache for SAN Architecture

For more details on the hardware configuration, see the Hardware configuration details section of this

white paper.


Test methodology

This section describes the test methodology when running the baseline and Fluid Cache analysis.

To show case the Dell Fluid Cache for SAN solution benefits, several benchmark tests were conducted

in the Dell Solutions lab simulating a production online transaction processing OLTP database. The

steps below provide a detailed understanding of the tests performed during the Dell Fluid Cache

performance study

An Oracle OLTP TPC-C workload is simulated using Quest®

Benchmark Factory

tool with a 10000 scale resulting in 1TB data being populated in the Oracle 12c

pluggable database.

A Compellent replay is taken on the database volumes after the data load as backup

and recovery strategy. This replay is used for restoring the data after each set of

OLTP Benchmark iteration to have consistent results.

Baseline benchmarking is performed on the two node cluster with a 1/50th

keying

and think-time values in the Benchmark factory tool to stress the database.

Fluid Cache for SAN software is enabled on all the three nodes and using

Compellent Enterprise Manager enabled the caching for the disks.

Fluid Cache for SAN benchmarking is then performed on the cache enabled disks

using the same 1/50th

keying values.

Keying Time creates a delay before a transaction executes, simulating activities such as data

entry that a user performs before executing the transaction.

Think Time creates a delay after a transaction executes, simulating a user’s decision-making

process before a user proceeds to the next transaction.

Figure 5: Cached Disks and Non-Cached Disks


Figure 6: Solution Design

Performance results and analysis

This section presents the performance results and analysis of the two solutions described in the

previous pages.

The baseline performance without caching provided a TPS of 449 and for the same configuration after

enabling the Fluid Cache software with a redundant cache of 700GB resulted in a significant number

of 1979 TPS. Baseline configuration cannot scale due to the storage disk bottleneck which resulted in

less number of TPS.

The Fluid Cache result provides a dramatic increase in TPS of 4.4x compared to the baseline results

Figure 7 shows a 4.4x improvement in TPS for Fluid Cache compared to the baseline.


Figure 7: OLTP Transactions Per Second

As shown in Figure 8, 1 second = 1000ms (milliseconds) response time is considered as the valid

benchmark threshold (SLA). The baseline configuration scaled upto 300 users with the response time

of 281 ms. The response time of the fluid cache solution is 29 ms at the same user load which

provides a 9.6x times better response time than the baseline configuration

Figure 8 shows a 9.6x improvement in response time for Fluid Cache compared to the baseline

configuration for an ART of 1 sec.

Figure 8: OLTP Average Response Time

0

500

1000

1500

2000

2500

10

0

30

0

50

0

70

0

90

0

11

00

13

00

15

00

17

00

19

00

TPS

User Load

OLTP Transactions/Sec (TPS)

Baseline

Fluid Cache

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

10

0

30

0

50

0

70

0

90

0

11

00

13

00

15

00

17

00

19

00

AR

T

User Load

OLTP Average Response Time (ART)

Baseline

Fluid Cache


The below analysis provides the maximum user load that the two solutions are able to deliver while

keeping the response time at one seconds or less SLA. The baseline user load scaled up to 500 users

and the Fluid Cache solution to 1900 users. So, Fluid Cache delivered ~4x more user load compared

to the baseline configuration at 1 sec SLA.

Figure 9 shows a ~4x increase in user load for Fluid Cache compared to the baseline.

Figure 9: User Load at One Second SLA

Conclusion

The Dell Fluid Cache solution brought data closer to compute by adding shared caching layer. It is a

best in class solution offering an optimal and cost effective way of accelerating application

performance. It supports global cache pool distributed across multiple servers, write-back cache

policies, and the easy access media that make it distinct from other host side caching solutions.

Essentially it enables customers to radically improve business by exceeding end user needs and

expectations. Dell Fluid Cache offers multiple features - boosted performance, easy to adopt, flexible

and scalable to implement etc., which make it a compelling choice for customers to adopt.

Dell Fluid Cache for SAN provides a reliable, efficient, and effective architecture, and key takeaways

are:

Boosted transactional performance while increasing concurrent users with low

latency.

Reduce response time by bringing data closer to compute.

Maximize computational horsepower with the Dell Fluid Cache for SAN solution

utilizing Dell PCI Express Flash drives.

Scale database performance on the fly with the addition of Express Flash drives.

Ensure data integrity with write-back SAN caching technology.

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Baseline Fluid Cache

Use

r Lo

ad

User Load at 1 sec SLA


Acknowledgement

I would like to thank the following Dell team members for providing significant support during

development and review of this whitepaper:

Balamurugan B, Makransky Karen, Phani MV, Reddy Mahesh, Reddy Ramamohan, Gopu, Narasimha

Reddy

References

Dell.com/Support is focused on meeting your needs with proven services and support.

DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell

employees for the purpose of sharing knowledge, best practices, and information about Dell products

and installations.

Referenced or recommended Dell publications:

How to deploy Oracle 12cR1 on RHEL6/Oracle Linux 6:

http://en.community.dell.com/techcenter/enterprise-

solutions/w/oracle_solutions/4960.how-to-deploy-oracle-12c-release-1-on-

rhel6oracle-linux-6.aspx

Oracle Best Practices on Compellent Storage Center

http://www.dellstorage.com/WorkArea/DownloadAsset.aspx?id=3055

Improving Oracle OLTP database performance with Dell Fluid

Cache for DAS

http://en.community.dell.com/techcenter/enterprise-

solutions/w/oracle_solutions/4615.improving-oracle-oltp-database-

performance-with-dell-fluid-cache-for-das.aspx

Referenced or recommended Oracle publications:

Oracle 12c documentation

http://docs.oracle.com/cd/E16655_01/server.121/e17906/chapter2.htm#NEWFT002

Oracle Database Storage Administrators Guide

http://download.oracle.com/docs/cd/B28359_01/server.111/b31107.pdf

Oracle Database Performance Tuning Guide

http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#PFGRF9522

9

http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/4960.how-to-deploy-oracle-12c-release-1-on-rhel6oracle-linux-6.aspx



http://www.dellstorage.com/WorkArea/DownloadAsset.aspx?id=3055

http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/4615.improving-oracle-oltp-database-performance-with-dell-fluid-cache-for-das.aspx



http://docs.oracle.com/cd/E16655_01/server.121/e17906/chapter2.htm#NEWFT002

http://download.oracle.com/docs/cd/B28359_01/server.111/b31107.pdf




Hardware configuration details

Server configuration

2 x Dell PowerEdge R820 servers (12th

generation) are used as database servers with the below components.

Component Description

Database server

2 X PowerEdge R820 BIOS: 2.0.19 iDRAC7 ESM FW: v1.46.45(Build 4) LC2: v 1.1.5.165

CPU 4 x Intel

® Xeon

® CPU E5-4657L v2 (12 cores per processor,

2.40GHz; 30MB cache)

Memory 512GB per server (32 X 16GB at 1600MT/s)

Fibre Channel interfaces QLogic QLE 2662, dual-port 16GB Fibre Channel HBA per server Driver version: 5.47 Firmware version: 03.30.19

Network interfaces

Broadcom 57800 2x10Gb SR/SFP+ + 2x1Gb BT network daughter card per server Driver version: 1.72.51-0 Firmware version: 7.6.15

Operating system disk

4x 146GB 15k SAS Disks in RAID10 Dell PERC H710p RAID controller Driver version: 06.504.01.00-rh1 Firmware version: 21.2.0-0007

PCIe SSDs 2* 350GB Dell PowerEdge Express Flash PCIe SSD, 2.5”

PCIe-SSD backplane PCIe-SSD extender 0 Firmware version 1.0

PCIe extender Adapter card

FW: 20.10.1-0084 Driver: 00.00.05.40-rh2 (native)

Mellanox adapter for Fluid Cache

MT27500 Family [ConnectX-3] Version 2.1.6 Firmware version: 02.30.51.10


1 x Dell PowerEdge R620 used as the Fluid Cache management server with the below components


Fluid Cache management server

1 X PowerEdge R620 BIOS: 2.0.19 iDRAC7 ESM FW: v1.35.35(Build 7) LC2: v 1.1.1.18

CPU 2 x Intel Xeon CPU E5-2660 (8 cores per processor, 2.20GHz ; 20MB cache)

Memory 64GB per server (4 X 16GB at 1600MT/s)

Fibre Channel Interfaces Qlogic QLE 2662, dual-port 16GB Fibre Channel HBA per server Driver version : 5.47 Firmware version: 03.30.19

Network interfaces Intel x 520 DP 1Gb BT +DP 10Gb SFP+ Rack NDC Driver version: 4.0.1-k Firmware version: Family 13.00.00

Operating system disk 4x 146GB 15k SAS disks in RAID10 Dell PERC H710p Mini Driver version 06.504.01.00-rh1 Firmware version 21.2.0-0007

Mellanox adapter for Fluid Cache MT27500 Family [ConnectX-3] Version 2.1.6 Firmware version: 02.30.51.10


Storage configuration

Storage is provided by a dual-controller Compellent array connected to a single SC220. Each

controller is deployed with 2 dual-port 16 Fibre Channel HBAs that serve front-end connectivity and

I/O. Backend connectivity is provided by a quad-port 6Gbps SAS controller per controller.

Component Description Storage controller 2x SC8000

Storage firmware revision Storage Center 6.5

Enclosure 1 X SC220 (24 x 2.5” drives) 20 X 300GB 15K SAS 19 + 1 (HS) disks 4 X 1 TB 7.2 SATA 3 + 1 (HS) disks

Backend connectivity 1 x quad-port 6Gb SAS per controller

Frontend connectivity Two dual-port 16Gb FC QLE 2662 HBA per controller

Network Switching Hardware

Below table provides the list of switching components used for SAN and LAN connectivity.


Fibre Channel switch

2 x Brocade 6510 Gen5 16Gb FC switches

Dell Fluid Cache switch

2 x Dell Networking S4810 switches

RAC cluster interconnect switch

2 x Dell Networking S4810 switches

Client IP switch 2 x Dell Networking S55 switches

Software configuration details

This section provides the software versions, software installation, and software configuration steps for

all the storage solutions described in the Solution design and reference architecture section.


Operating system RHEL 6.4 Kernel 2.6.32-358.el6.x86_64 (x86_64)

Database type Oracle 12cR1 (12.1.0.1.0) 2-node RAC


Dell Fluid Cache for SAN v 1.0.0

Database size 1.2TB deployed on a pluggable database

ASM Yes – with external redundancy

Oracle System Global Area (SGA)

60GB per node

Load generator Quest BenchMark Factory TPC-C style workload

Storage management software

Compellent Enterprise Manager Version : 6.5

Oracle configuration settings

Kernel parameter settings

kernel.msgmnb = 65536

kernel.msgmax = 65536

fs.file-max = 6815744

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

net.core.rmem_max = 4194304

net.core.rmem_default = 262144

kernel.shmmni = 4096

kernel.shmmax = 274877906944

kernel.shmall = 67108864

kernel.sem = 250 32000 100 128


User Security Limits Settings

oracle soft nofile 1024

oracle hard nofile 65536

oracle soft nproc 131072

oracle hard nproc 131072

oracle soft stack 10240

oracle hard stack 32768

grid hard nofile 131072

grid soft nproc 131072

* soft memlock unlimited

* hard memlock unlimited

Database parameter settings

db_writer_processes 4

processes 10000

sessions 15072

open_cursors 10000

transactions 10000

undo_retention 10800

Udev settings Settings for Baseline Configuration

This section describes the udev rules that need to be set for the Baseline solution.

1. It is recommended to set the udev rules for the block devices to be used for ASM disk using their WWIDs. Identify the WWIDs for all the virtual disks that are to be configured for Oracle ASM by running the following command

$> scsi_id --page=0x83 --whitelisted --device=/dev/sdX

where sdX is the name of the block device to be used for Oracle


Example output: sdc 360026b900061855e000008a54ea5356a

2. Edit the /etc/udev/rules.d/99-Dell_Oracle.rules file (create a new file if it doesn’t already

exist) and add the following lines to set the permissions needed for the baseline

configuration. Replace each of the <WWID_for_sdX> below with the correct equivalent

WWID result obtained from step 1 for that sdX block device.

KERNEL=="dm*", PROGRAM="scsi_id --page=0x83 --whitelisted --

device=/dev/%k", RESULT=="36000d31000ece2000000000000000075",

NAME="fldc0", OWNER="grid", GROUP="asmadmin", MODE="0660"







Udev Settings for Fluid Cache Configuration

For setting the permissions for the Fluid Cache disks, edit the /etc/udev/rules.d/99-Dell_Oracle.rules

file as below:

KERNEL=="fldc0", OWNER:="grid", GROUP:="asmadmin", MODE="0660"



Server Configuration Profile

Dell iDRAC7 with Lifecycle Controller provides the ability to generate a human readable snapshot of

server configuration using the XML feature. This single file contains all BIOS, iDRAC, LC, Network, and

Storage settings. The file captured from one server can be applied to other servers, even across

different server models.


Configuration XML operations are performed via both RACADM and WS-MAN commands and can be

directed to or from network shares (NFS / CIFS).

The Improving Oracle OLTP 12c database performance with Dell™ Fluid Cache for SAN whitepaper

also provides a server configuration profile (xml) to directly import the Dell Engineered configuration

into Dell servers.

You can download the configuration profile here.

http://en.community.dell.com/techcenter/enterprise-solutions/m/oracle_db_gallery/20439120/download.aspx

accelerating oracle oltp 12c database performance with...

Documents