huawei oceanstor v3 converged storage systems … systems — oltp oracle database reference...
TRANSCRIPT
HUAWEI OceanStor V3 Converged
Storage Systems — OLTP Oracle
Database Reference Architectures
HUAWEI OceanStor V3 converged storage systems are next-generation storage products intended for enterprise
customers. How to select proper storage models and configurations based on the database scale and performance
requirements for the maximum return on investment (ROI) is a challenge faced by customers and Huawei. The
solutions described in this document verify OLTP Oracle Database of several common scales, providing reference
about OLTP databases for enterprise customers.
The solutions use OceanStor 5300 V3, 5500 V3, 5600 V3, and 5800 V3 converged storage systems to respectively
provide at least 20,000, 40,000, 60,000, and 80,000 transaction IOPS for OLTP Oracle 12c RACs whose scales are 2
TB, 4 TB, 6 TB, and 8 TB respectively.
Based on Oracle 12c clusters and pluggable database architectures, the solutions use RH2288 servers as database
nodes and are oriented to enterprise databases that require high transaction IOPS.
Wang Yaohui
Storage Solutions, IT, Huawei Enterprise BG
2015-03-17 V1.0
Huawei Technologies Co., Ltd.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 2
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Contents
1 Overview ......................................................................................................................................... 4
1.1 Introduction .................................................................................................................................................................. 4
1.2 Solution Architectures .................................................................................................................................................. 4
1.3 Intended Audience ........................................................................................................................................................ 8
1.4 Customer Benefits ........................................................................................................................................................ 9
1.5 Key Components .......................................................................................................................................................... 9
1.6 Workload Model ........................................................................................................................................................... 9
2 Products and Technologies ....................................................................................................... 10
2.1 OceanStor V3 Converged Storage Systems ................................................................................................................ 10
2.1.1 Next-Generation Hardware ...................................................................................................................................... 11
2.1.2 Multi-Controller Architecture .................................................................................................................................. 11
2.1.3 Convergence Design ................................................................................................................................................ 11
2.1.4 Smart Software ........................................................................................................................................................ 11
2.1.5 Unified and Easy Management ................................................................................................................................ 12
2.1.6 RAID 2.0+ Block Virtualization .............................................................................................................................. 12
2.1.7 SmartTier ................................................................................................................................................................. 13
2.2 Oracle Database and Cluster ....................................................................................................................................... 15
2.2.1 Oracle RAC and ASM ............................................................................................................................................. 15
2.2.2 Oracle System Architecture ..................................................................................................................................... 17
2.2.3 Oracle Application Types ......................................................................................................................................... 19
3 Tiny-Size OLTP Database Reference Architecture .............................................................. 21
3.1 Huawei Solution ......................................................................................................................................................... 21
3.1.1 Solution Architecture ............................................................................................................................................... 21
3.1.2 Solution Configuration ............................................................................................................................................ 22
3.2 Verification Procedure ................................................................................................................................................ 24
3.3 Verification Results ..................................................................................................................................................... 27
4 Small-Size OLTP Database Reference Architecture ............................................................ 28
4.1 Huawei Solution ......................................................................................................................................................... 28
4.1.1 Solution Architecture ............................................................................................................................................... 28
4.1.2 Solution Configuration ............................................................................................................................................ 29
4.2 Verification Procedure ................................................................................................................................................ 31
4.3 Verification Results ..................................................................................................................................................... 31
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 3
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
5 Medium-Size OLTP Database Reference Architecture ....................................................... 33
5.1 Huawei Solution ......................................................................................................................................................... 33
5.1.1 Solution Architecture ............................................................................................................................................... 33
5.1.2 Solution Configuration ............................................................................................................................................ 34
5.2 Verification Procedure ................................................................................................................................................ 36
5.3 Verification Results ..................................................................................................................................................... 37
6 Large-Size Transaction Database Reference Architecture .................................................. 38
6.1 Huawei Solution ......................................................................................................................................................... 38
6.1.1 Solution Architecture ............................................................................................................................................... 38
6.1.2 Solution Configuration ............................................................................................................................................ 39
6.2 Verification Procedure ................................................................................................................................................ 41
6.3 Verification Results ..................................................................................................................................................... 42
7 Appendix ...................................................................................................................................... 43
7.1 Reference Documents ................................................................................................................................................. 43
7.2 Acronyms and Abbreviations ...................................................................................................................................... 43
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 4
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
1 Overview
1.1 Introduction
HUAWEI OceanStor V3 converged storage systems are next-generation storage products
intended for enterprise customers. How to select proper storage models and configurations
based on the database scale and performance requirements for the maximum return on
investment (ROI) is a challenge faced by customers and Huawei. The solutions described in
this document verify OLTP Oracle Database of several common scales, providing reference
about OLTP databases for enterprise customers.
The solutions use OceanStor 5300 V3, 5500 V3, 5600 V3, and 5800 V3 converged storage
systems to respectively provide at least 20,000, 40,000, 60,000, and 80,000 transaction IOPS
for OLTP Oracle 12c clusters whose scales are 2 TB, 4 TB, 6 TB, and 8 TB respectively.
Based on Oracle 12c clusters and pluggable database architectures, the solutions use RH2288
servers as database nodes and are oriented to enterprise databases that require high transaction
IOPS.
1.2 Solution Architectures
The following table describes the verified reference architectures.
Table 1-1 Reference architecture list
Solution Component Amount of Data Transaction IOPS
T (tiny-size
OLTP database)
OceanStor 5300 V3
converged storage
system
2 x RH2288 V2 servers
2 TB
(Lab test data: 2.5
TB of table and
index data)
20,000
(Lab test data:
28,877)
S (small-size
OLTP database)
OceanStor 5500 V3
converged storage
system
4 x RH2288 V2 servers
4 TB
(Lab test data: 5 TB
of table and index
data)
40,000
(Lab test data:
44,172)
M (medium-size
OLTP database)
OceanStor 5600 V3
converged storage
system
6 x RH2288 V2 servers
6 TB
(Lab test data: 8 TB
of table and index
data)
60,000
(Lab test data:
63,477)
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 5
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Solution Component Amount of Data Transaction IOPS
L (large-size
OLTP database)
OceanStor 5800 V3
converged storage
system
8 x RH2288 V2 servers
8 TB
(Lab test data: 10
TB of table and
index data)
80,000
(Lab test data:
86,429)
Figure 1-1 Tiny-size OLTP database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5300 V3 converged storage system is as follows: 32 GB
cache, 1 x controller enclosure (18 x 600 GB 10k rpm SAS disks, 7 x 200 GB SLC SSDs, 2 x 8 Gbit/s
four-port Fibre Channel I/O modules), 1 x disk enclosure (25 x 600 GB 10k rpm SAS disks), 1 x disk
domain containing all the 50 disks (22.7 TB capacity), 1 x storage pool (RAID 10 and 6 TB capacity
configured for the SAS tier, and RAID 5-5 and 800 GB capacity configured for the SSD tier), 10 x 500
GB LUNs (eight LUNs used as the data area, and two LUNs used as the log area).
[3] The tested database contains 2.5 TB of table and index data.
[4] The tested transaction IOPS is 28,877, based on the Order Entry 2.0 order processing model.
5300 V3 [2]
43 x 600 GB 10k rpm SAS disks7 x 200 GB SLC SSDs
2 x
12
Gb
it/s
SA
S
4 x 8 Gbit/s FC
1 x disk enclosureOracle 12c RAC
2 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Tiny-Size OLTP Database Solution2 TB[3] data, 20,000[4] transaction IOPS
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 6
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Figure 1-2 Small-size transaction database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5500 V3 converged storage system is as follows: 48 GB
cache, 1 x controller enclosure (13 x 600 GB 10k rpm SAS disks, 12 x 200 GB SLC SSDs, 2 x 8 Gbit/s
four-port Fibre Channel I/O modules), 3 x disk enclosures (75 x 600 GB 10k rpm SAS disks), 1 x disk
domain containing all the 100 disks (46.1 TB capacity), 1 x storage pool (RAID 10 and 12 TB capacity
configured for the SAS tier, and RAID 5-9 and 1600 GB capacity configured for the SSD tier), 20 x 500
GB LUNs (16 LUNs used as the data area, and 4 LUNs used as the log area).
[3] The tested database contains 5 TB of table and index data.
[4] The tested transaction IOPS is 44,172, based on the Order Entry 2.0 order processing model.
5500 V3 [2]
88 x 600 GB 10k rpm SAS disks12 x 200 GB SLC SSDs
4 x
12
Gb
it/s
SA
S
4 x 8 Gbit/s FC
3 x disk enclosures
Oracle 12c RAC
4 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Small-Size OLTP Database Solution4 TB[3] data, 40,000[4] transaction IOPS
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 7
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Figure 1-3 Medium-size OLTP database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5600 V3 converged storage system is as follows: 64 GB
cache, 1 x controller enclosure (2 x 8 Gbit/s four-port Fibre Channel I/O modules, 2 x 12 Gbit/s
four-port SAS I/O modules), 6 x disk enclosures (132 x 600 GB 10k rpm SAS disks, 18 x 200 GB SLC
SSDs), 1 x disk domain containing all the 150 disks (69.1 TB capacity), 1 x storage pool (RAID 10 and
18 TB capacity configured for the SAS tier, and RAID 5-9 and 2400 GB capacity configured for the
SSD tier), 30 x 500 GB LUNs (24 LUNs used as the data area, and 6 LUNs used as the log area).
[3] The tested database contains 8 TB of table and index data.
[4] The tested transaction IOPS is 63,477, based on the Order Entry 2.0 order processing model.
6 x
12 G
bit/s
SA
S
4 x 8 Gbit/s FC
6 x disk enclosures
Oracle 12c RAC
6 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Medium-Size OLTP Database Solution6 TB[3] data, 60,000[4] transaction IOPS
5600 V3 [2]
132 x 600 GB 10k rpm SAS disks18 x 200 GB SLC SSDs
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 8
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Figure 1-4 Large-size transaction database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5800 V3 converged storage system is as follows: 128
GB cache, 1 x controller enclosure (1 x 8 Gbit/s four-port Fibre Channel I/O module, 2 x 12 Gbit/s
four-port SAS I/O modules), 8 x disk enclosures (175 x 600 GB 10k rpm SAS disks, 25 x 200 GB SLC
SSDs), 1 x disk domain containing all the 200 disks (91.8 TB capacity), 1 x storage pool (RAID 10 and
24 TB capacity configured for the SAS tier, and RAID 5-9 and 3200 GB capacity configured for the
SSD tier), 40 x 500 GB LUNs (32 LUNs used as the data area, and 8 LUNs used as the log area).
[3] The tested database contains 10 TB of table and index data.
[4] The tested transaction IOPS is 86,429, based on the Order Entry 2.0 order processing model.
1.3 Intended Audience
This document is intended for Huawei partners and customers as well as storage and database
administrators and IT engineers who want to deploy OLTP Oracle Database 12c based on
HUAWEI OceanStor V3 converged storage systems.
It is assumed that the readers are familiar with the following products and technologies:
8 x
12 G
bit/s
SA
S
Oracle 12c RAC
8 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Large-Size OLTP Database Solution8 TB[3] data, 80,000[4] transaction IOPS
8 x disk enclosures
5800 V3 [2]
175 x 600 GB 10k rpm SAS disks25 x 200 GB SLC SSDs
4 x 8 Gbit/s FC
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 9
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Architecture and working principles of OceanStor V3 converged storage systems
Architecture and working principle of Oracle Database 12c
Linux operating system basics
1.4 Customer Benefits
The solutions described in this document are designed to accelerate the process of designing,
verifying, and delivering an OLTP database solution. Based on a typical OLTP model, this
document provides verified solutions that use OceanStor V3 converged storage systems.
Typical configurations and performance indicators are provided for reference, simplifying
selection of storage models as well as storage planning and configuration. The verified
solutions are expected to help customers obtain the maximum ROI.
1.5 Key Components
The hardware and software covered in this document are as follows:
Storage system: OceanStor V3 converged storage system V300R001C10
Operating system: Red Hat Enterprise Linux 6.5
Multipathing software: UltraPath for Linux 8.01.024
Database software: Oracle Database 12.1.0.2
Cluster software: Oracle RAC 12.1.0.2
Test tool: Huawei SwingBench Test Suite 1.0 for Oracle 12c
1.6 Workload Model
This document uses the mainstream OLTP test model SwingBench Order Entry to test the
recommended planning and configuration strategies. This model defines a type of online order
business and simulates a scenario where a large number of users are querying products,
placing orders, processing orders, and viewing orders online. Those operations are the most
common ones in transaction systems. In this workload model, there are two main performance
indicators: transactions per second (TPS) and average transaction response time. The TPS
indicates the number of transactions processed per second. A higher TPS indicates higher
productivity. The average transaction response time directly affects the speed of user
operations. A shorter response time indicates better user experience.
Order Entry defines nine tables, storing information about products, customers, orders,
warehouses, and login. During the workload tests, 50% of operations are SELECT, 30%
INSERT, 20% UPDATE, and no DELETE operations. From the perspective of I/O layer, the
workload model is the most typical OLTP workload model, where small data blocks are
accessed at random and the ratio between reads and writes is 6:4.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 10
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
2 Products and Technologies
2.1 OceanStor V3 Converged Storage Systems
HUAWEI OceanStor V3 converged storage systems are next-generation unified storage
products designed for enterprise-class applications. Leveraging a storage operating system
oriented to cloud architecture, a powerful next-generation hardware platform, and a full range
of intelligent management software, OceanStor V3 converged storage systems deliver
industry-leading functionality, performance, efficiency, reliability, and ease of use. They
provide data storage for applications such as large-scale database OLTP/OLAP, file sharing,
and cloud computing, and can be used in industries ranging from government, finance,
telecommunications, energy, to media and entertainment (M&E). Meanwhile, OceanStor V3
converged storage systems can provide a wide range of efficient and flexible backup and
disaster recovery solutions to ensure business continuity and data security, delivering
excellent storage services.
For details about HUAWEI OceanStor V3 converged storage systems, click the following
link:
http://e.huawei.com/en/products/cloud-computing-dc/storage/unified-storage/mid-range
Figure 2-1 HUAWEI OceanStor V3 converged storage systems
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 11
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
2.1.1 Next-Generation Hardware
OceanStor V3 converged storage systems employ next-generation Intel multi-core processors,
PCIe 3.0 buses, 12 Gbit/s SAS 3.0 high-speed disk ports, and a variety of host ports such as
16 Gbit/s Fibre Channel, 10 Gbit/s FCoE, and 56 Gbit/s InfiniBand host ports. The storage
systems provide up to 28 GB/s of system bandwidth to meet the requirements of
bandwidth-intensive application scenarios. They also offer million-level IOPS performance,
outshining products from other vendors.
OceanStor V3 converged storage systems are equipped with exclusive SmartIO cards. A
SmartIO card supports 8 Gbit/s Fibre Channel, 16 Gbit/s Fibre Channel, 10 Gbit/s iSCSI, and
10 Gbit/s FCoE. Users can specify the protocols that a SmartIO card is required to support.
The deduplication/compression cards used by OceanStor V3 converged storage systems
support lossless data deduplication and compression, efficiently reducing data storage costs.
In addition, the storage systems can implement data encryption to secure data.
2.1.2 Multi-Controller Architecture
The multi-controller architecture used by OceanStor V3 converged storage systems supports
online horizontal expansion. An OceanStor V3 converged storage system can be
non-disruptively expanded to a maximum of eight controllers, 1 TB of cache, and 5 TB of
storage space, meeting customers' future capacity needs. The multi-controller architecture
allows load balancing among controllers and eliminates single points of failure, thereby
ensuring high availability and stable service running.
2.1.3 Convergence Design
Convergence of SAN and NAS: SAN and NAS services are converged to provide
elastic storage, simplify service deployment, improve storage resource utilization, and
reduce the total cost of ownership (TCO). Underlying storage resource pools directly
provide both block and file services, thereby shortening storage resource access paths to
ensure that the two services are equally efficient.
Convergence of heterogeneous storage systems: Based on the built-in heterogeneous
virtualization function, OceanStor V3 converged storage systems can efficiently manage
storage systems from other mainstream vendors and unify resource pools for central and
flexible resource allocation.
Convergence of entry-level, mid-range, and high-end storage systems: OceanStor V3
converged storage systems are the only storage systems in the industry that enable
entry-level, mid-range, and high-end storage systems to interwork seamlessly with each
other. Data can freely flow among storage products of different models without the
assistance of third-party systems.
Convergence of SSDs and HDDs: The advantages of traditional and solid-state storage
media are combined, bringing the performance of different types of storage media into
full play and striking an optimal balance between performance and cost.
Convergence of primary and backup storage: The built-in backup function enables
data to be efficiently backed up without additional backup software, simplifying backup
solution management.
2.1.4 Smart Software
Multi-tenancy and service level agreement (SLA): OceanStor V3 converged storage
systems intelligently allocate storage resources in cloud computing environments to meet
the needs of enterprises and organizations. The storage systems also leverage data
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 12
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
isolation and a range of data security policies such as data encryption and data
destruction to meet varying data security requirements. OceanStor V3 converged storage
systems provide four service levels and allocate storage resources based on service
priorities. Storage resources are first allocated to high-priority services to ensure system
performance and shorten response time.
Smart-series efficiency improvement suite: OceanStor V3 converged storage systems
use SmartTier (dynamic storage tiering), SmartMotion (intelligent data migration), and
innovative SmartVirtualization (heterogeneous virtualization) to achieve vertical,
horizontal, and cross-system 3D data flowing, improving storage resource utilization by
three times.
Hyper-series data protection software: Data protection software such as remote
replication, snapshot, and LUN copy software meets user needs for local, remote, and
multi-region data protection, maximizing business continuity and data availability.
2.1.5 Unified and Easy Management
Unified management: OceanStor V3 converged storage systems provide powerful
storage management software that supports global topology view, capacity analysis,
performance analysis, fault diagnosis, and end-to-end service visualization to manage a
wide range of devices.
Convenient management: OceanStor V3 converged storage systems can be initially
configured in 5 steps which take about 40 seconds, and expanded in two steps which take
about 15 seconds.
Mobile management: Users can use tablets and mobile phones to manage storage
systems in real time. System status is sent automatically, making constant attendance by
an engineer unnecessary.
2.1.6 RAID 2.0+ Block Virtualization
RAID 2.0+ block virtualization of the OceanStor V3 implements virtualization for underlying
disk management and upper-layer resource management. Inside the system, the storage space
of each disk is divided into fine-grained data blocks, which comprise RAID groups. In doing
so, data is evenly distributed to all disks in the storage pool. In addition, data block–based
resource management largely improves the resource management efficiency.
1. The V3 storage systems support SSDs, SAS disks, and NL-SAS disks. These disks
comprise disk domains. In a disk domain, disks of the same type comprise disk groups
(DGs).
2. In a DG, the storage space of disks is divided into chunks (CKs) of a fixed size. Then the
system consolidates CKs from random disks into CK groups (CKGs) based on RAID
algorithms.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 13
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
3. CKGs are divided into logical storage space called extents, which have a fixed size too.
Extents are the minimum unit for comprising thick LUNs. On thin LUNs, extents are
further divided into smaller grains.
2.1.7 SmartTier
SmartTier is a DST feature independently developed by Huawei based on RAID 2.0+. It
creates different storage tiers based on disk types, analyzes the activity levels of data blocks,
and migrates data among tiers based on analysis results. This feature ensures that data is
stored in proper storage media, improving system performance and reducing the TCO.
OceanStor V3 converged storage systems support three types of disks, namely, SSDs, SAS
disks, and NL-SAS disks. Each type of storage medium has its unique advantages and
disadvantages in performance and cost. As a result, it is hard for customers to strike a balance
between storage costs and storage performance.
SSDs: Feature a short response time, a low storage request processing cost, but a high
storage capacity cost per gigabyte.
NL-SAS disks: Have a high request processing cost but a low capacity cost per gigabyte.
SAS disks: Fall in between the previous two types in terms of performance and cost.
To strike a balance among storage performance, capacity, and cost, SmartTier incorporates the
following advantages:
High performance and multi-application support: various applications and differentiated
application performance demands
Flexibility and ease-of-use: convenient configuration and migration of data to a suitable
tier
High efficiency and low power consumption: energy-saving and efficient use of space
Table 2-1 Properties of the three tiers in SmartTier
Tier Disk Type Application
High-performa
nce tier
SSD Applicable to applications with intensive
random access requests
Performance
tier
SAS Applicable to applications with medium access
requests
Capacity tier NL-SAS Applicable to applications with low access
requests but high capacity requirements
As shown in the following figure, SmartTier performs intelligent data storage at a LUN level and divides data on LUNs based on the default data migration granularity of 4 MB. The data
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 14
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
migration granularity is called extent and ranges from 512 KB to 64 MB. SmartTier collects
statistics and analyzes data activity levels based on extents, and matches data of different
activity levels with storage media. Active data will be promoted to higher-performance
storage media (such as SSDs), whereas inactive data will be demoted to more cost-effective
storage media with larger capacities (such as NL-SAS disks).
Figure 2-2 Working principle of SmartTier
A complete SmartTier service process involves three phases:
Phase I: Collecting hotspot statistics
SmartTier allows user-defined I/O monitoring periods. During the scheduled periods, it
collects I/O statistics. Activity levels of data will change throughout a data lifecycle. By
comparing the activity level of one extent with that of another, the storage system
determines which data block is more or less frequently accessed. The activity level of
each extent is obtained based on the performance indicator statistics of data blocks.
Phase II: Analyzing data placement
The collected hotspot statistics are analyzed. This analysis ranks extents within the
storage pool. The ranking progresses from the most frequently accessed extents to the
least frequently accessed extents in the same storage pool. Note that only extents in the
same storage pool are ranked. Then a data migration solution is created. Before data
migration, SmartTier determines the direction of migrating extents according to the latest
data migration solution.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 15
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Phase III: Migrating data
SmartTier has two migration triggering modes: manual and periodic. The manual
triggering mode has a higher priority than the periodic mode. In manual triggering mode,
data migration can be triggered immediately when necessary. In periodic triggering mode,
data migration is automatically triggered based on a preset migration start time and
duration. The start time and duration of data migration are user-definable.
2.2 Oracle Database and Cluster
Oracle Database is one of the most widely used relational databases. This section briefly
introduces Oracle Database 12c and focuses on Multitenant-related components and features,
including RAC, ASM, Multitenant, data files, database instance architecture, and application
types.
2.2.1 Oracle RAC and ASM
As shown in Figure 2-3, an Oracle 12c RAC contains two types of nodes: Hub nodes and Leaf
nodes. Hub nodes have direct access to shared storage, whereas Leaf nodes access shared
storage through Hub nodes. When a database is deployed on an Oracle RAC, the nodes can be
grouped into multiple server pools. Each database is deployed in a server pool, and every
node in a server pool runs a database instance. Application servers access the virtual IP
addresses (VIPs) of nodes to store data. If a node fails, its VIP network is restored on another
node of Oracle RAC. Application servers reconnect to the Oracle database through a
reconnection mechanism. Setting connection character strings on application servers can
enable multiple modes of accessing Oracle RAC nodes, including load balancing and failover
modes. In these modes, a multi-node Oracle cluster is presented as a single database to
application servers.
The shared storage of the Oracle RAC Hub nodes includes Oracle Cluster Registry (OCR),
voting disks, and database. OCR records information about node statuses, voting disks
synchronize data between nodes, and the database is a set of files.
Figure 2-3 Oracle Flex Cluster 12c
Oracle ASM provides a simple storage management interface for database administrators to
manage servers and storage across different platforms. As a built-in file system and volume manager, Oracle ASM is exclusive to Oracle database files. ASM simplifies file system
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 16
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
management, provides asynchronous I/O performance tuning, saves management time for
administrators, and offers a flexible, efficient database environment.
ASM can consolidate LUNs into a disk group and use Allocation Units (AUs) to allocate
storage space from the disk group. ASM supports three types of disk groups.
External: Data is not mirrored between LUNs, and the storage system provides data
protection.
Normal: A normal disk group consists of two failure groups between which data is
mirrored.
High: A high disk group consists of three failure groups among which data is mirrored.
When an OceanStor V3 storage system is used to create ASM disk groups, it is recommended
that controllers A and B evenly own the LUNs in the disk groups before external or normal
disk groups are created.
Oracle Flex ASM is a new Oracle ASM deployment model that increases database instance
availability and reduces Oracle ASM related resource consumption. Oracle Flex ASM
facilitates cluster based database consolidation, as it ensures that Oracle Database 12c
instances running on a particular server will continue to operate, should the Oracle Flex ASM
instance on that server fail.
Figure 2-4 ASM before Oracle Database 12c
Figure 2-5 Oracle 12c Flex ASM
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 17
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Oracle RAC provides the following key characteristics, essential for HA data management:
Reliability — Oracle Database is known for its reliability. Oracle RAC takes this step
further by removing the database server as a single point of failure. If an instance fails,
the remaining instances in the cluster remain open and active. Oracle Clusterware
monitors all Oracle processes and immediate restarts any failed component.
Error detection — Oracle Clusterware automatically monitors all Oracle RAC databases
as well as other Oracle processes (Oracle ASM, instances, Listeners, etc.) and provides
fast detection of problems in the environment. It also automatically recovers from
failures often before users notice that a failure has occurred.
Recoverability — The Oracle Database includes many features that make it easy to
recover from various types of failures. If an instance fails in an Oracle RAC database, it
is recognized by another instance in the cluster and recovery will start automatically. Fast
Application Notification (FAN) and Fast Connection Failover (FCF) and especially the
Oracle RAC 12c Application Continuity feature make it easy to mask any component
failure from the user.
Continuous Operations — Oracle RAC provides continuous service for both planned and
unplanned outages. If a server (or an instance) fails, the database remains open and
applications continue to be able to access data, allowing for business critical workloads
to finish, mostly without a delay in service delivery.
For more information about Oracle 12c RAC and ASM, refer to the following documents:
White Paper: Oracle Real Application Clusters (RAC)
A Technical Overview of New Features for Automatic Storage Management in Oracle
Database 12c
2.2.2 Oracle System Architecture
Figure 2-6 Oracle system architecture
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 18
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
As shown in the preceding figure, System Global Area (SGA) and Program Global Area
(PGA) of Oracle databases consume memory. SGA stores system information and page cache
information, and PGA stores session information. SGA contains the following parts:
Buffer Cache: buffers data blocks.
Redo Log Buffer: buffers log records as a recycle data group.
Share Pool: buffers data dictionaries and shared SQL information.
Oracle files are categorized as follows:
Control file: records the database structure, parameters, and locations of other data fails.
Data file: stores user data and temporary data.
Online log: record changes to data blocks and consists of several log groups. Files in the
log groups are mirrored to each other. After a log group is used up, data in written to the
next log group. After the first log group is used up, data is written to the first log group
again.
Archive log: In archive mode, databases copy fully written log groups to the archive area
for data restoration when anomalies occur.
Among Oracle processes, the Listener process monitors client connections. Clients are
connected in two modes:
In shared mode, the listener redirects client requests to the dispatcher process, which
places the request in the request queue of the large pool. Then the shared server obtains
and processes the request in the request queue and puts the processing results in the
response queue of the large pool. At last, the dispatcher returns the processing results to
the client.
In dedicated mode, a dedicated server process serves each client connection. After
receiving the request from a client, the Oracle server process looks for the data block in
the buffer cache. If the data block is found, data is read, computed, and changed in the
buffer cache. If the data block does not exist in the buffer cache, the Oracle server
process writes the data block from data files to the buffer cache, and then computes and
changes it.
Oracle uses the LRU algorithm to eliminate outdated data in the buffer cache so that the
released storage space can be used by new data blocks. Data that has been changed in the
buffer cache is called "dirty data", which is written by the DB writer (DBW) process to data
files.
To ensure data integrity and reliability, relational databases use "transaction" to indicate an
atomic operation. When processing a transaction, the Oracle server process records changed
data and the change time in the log buffer. When the transaction is delivered, the log writer
(LGWR) synchronizes the log data in the log buffer to online log files. The log buffer is a
memory area where data can be written in a circular manner. When the log buffer is one-third
full, the LGWR synchronizes the log records to the log file, regardless of whether the
transaction is committed. In addition, Oracle databases synchronize logs automatically every
three minutes.
By default, Oracle databases perform a checkpoint every 30 minutes. When the checkpoint is
performed, the Checkpoint (CKPT) process synchronizes the control file and triggers the
DBW to write dirty data to data files.
Online logs of Oracle are divided into several groups, each of which contains one or multiple
log files. When multiple log files exist, the files are mirrored to each other. Oracle databases
write logs to the log groups in sequence. When the last log group is filled, the databases write
to the first log group, restarting the cycle. Before a log group switch, Oracle checks whether
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 19
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
the dirty data recorded in the next log group is completely written to data files. If not, Oracle
waits until the DBW process writes all the dirty data to the data files before it starts the log
group switch.
When an Oracle database is in archive mode, the Archive (ARC) process copies filled logs to
the archive area. If data anomalies occur, the archived logs are used for precise data recovery.
Oracle Database 12c also supports a Multitenant that allows multiple PDBs to run in one
CDB. Figure 2-7 shows a CDB with four containers: the root, seed, and two PDBs (hrpdb and
salespdb). Each PDB has its own dedicated application. A different PDB administrator
manages each PDB. User SYS can manage the root and every PDB. At the physical level, this
CDB has a database instance and database files. The Multitenant feature brings better
serviceability to the Oracle Database.
The seed PDB is a system-supplied template that the CDB can use to create new PDBs. The seed PDB is
named PDB$SEED. You cannot add or modify objects in PDB$SEED.
Figure 2-7 Oracle Multitenant architecture
For more information about Oracle Multitenant Architecture, refer to the following document:
White Paper: Oracle Multitenant
2.2.3 Oracle Application Types
Data transactions are categorized as two types: OLTP and OLAP.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 20
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
OLTP: A number of online users perform transactions.
OLAP: A small number of users perform long-term complex statistical queries.
OLTP applications have the following I/O characteristics:
From the perspective of database
The reading, writing, and changing of each transaction involve a small amount of data.
Database data must be up-to-date. Therefore, OLTP applications require a high database
availability.
Many users connect to and use the database concurrently.
The database must be highly responsive and able to complete a transaction within
seconds.
From the perspective of storage
Every I/O is small-sized, ranging from 2 KB to 8 KB.
Disk data is randomly accessed.
At least 30% of data is generated by random writing operations.
Redo logs are written frequently.
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 21
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
3 Tiny-Size OLTP Database Reference Architecture
3.1 Huawei Solution
3.1.1 Solution Architecture
Figure 3-1 Tiny-size OLTP database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5300 V3 converged storage system is as follows: 32 GB
cache, 1 x controller enclosure (18 x 600 GB 10k rpm SAS disks, 7 x 200 GB SLC SSDs, 2 x 8 Gbit/s
four-port Fibre Channel I/O modules), 1 x disk enclosure (25 x 600 GB 10k rpm SAS disks), 1 x disk
domain containing all the 50 disks (22.7 TB capacity), 1 x storage pool (RAID 10 and 6 TB capacity
configured for the SAS tier, and RAID 5-5 and 800 GB capacity configured for the SSD tier), 10 x 500
GB LUNs (eight LUNs used as the data area, and two LUNs used as the log area).
5300 V3 [2]
43 x 600 GB 10k rpm SAS disks7 x 200 GB SLC SSDs
2 x
12
Gb
it/s
SA
S
4 x 8 Gbit/s FC
1 x disk enclosureOracle 12c RAC
2 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Tiny-Size OLTP Database Solution2 TB[3] data, 20,000[4] transaction IOPS
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 22
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[3] The tested database contains 2.5 TB of table and index data.
[4] The tested transaction IOPS is 28,877, based on the Order Entry 2.0 order processing model.
3.1.2 Solution Configuration
Table 3-1 Hardware configuration
Solution Hardware Component Quantity
Tiny-Size OLTP
Database
Solution
Server: RH2288
V2
2
Intel® Xeon
® E5-2660 @ 2.20 GHz 2
Memory 256 GB
QLogic 8 Gbit/s dual-port Fibre
Channel HBA
1
Intel 10 Gbit/s dual-port Ethernet
HBA
1
Storage:
OceanStor 5300
V3
1
Cache 32 GB
2 U controller enclosure with 25 slots 1
2 U disk enclosure with 25 slots 1
600 GB 10k rpm SAS disk 43
200 GB SLC SSD 7
8 Gbit/s four-port Fibre Channel I/O
module
2
Fibre Channel
switch: SNS2224
2
Private network
switch: S6700
1
Table 3-2 Software configuration
Solution Hardware Software Quantity
Tiny-Size OLTP
Database
Solution
RH2288 V2 2
Operating system: Red Hat Enterprise
Linux 6.5
Multipathing software: UltraPath for
Linux 8.01.024
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 23
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Solution Hardware Software Quantity
Database cluster: Oracle Grid 12.1.0.2
Database software: Oracle Database
12.1.0.2
Test tool: Huawei SwingBench Test
Suite 1.0 for Oracle 12c
OceanStor
5300 V3
1
SmartTier license
SmartThin license
SmartMotion license
SmartQoS license
Figure 3-2 Storage configuration
[1] The disk domain configuration is as follows: The hot spare policy of the SAS tier is High (default
policy), and that of the SSD tier is Low.
[2] The storage pool configuration is as follows: RAID 10 and 6 TB capacity are configured for the SAS
tier. RAID 5-5 and 800 GB capacity are configured for the SSD tier. The SmartTier monitoring period is
from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day. Other
parameters keep the default values.
[3] The LUN configuration of the data area is as follows: The SmartTier policy is automatic migration.
LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
The LUN configuration of the log area is as follows: The read/write policy is reclamation. The priority is
high. LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
Disk
Disk Domain [1]
LUN [3]
ASM Disk Group [4]+DATA +LOG+GRID
…43 x 600 GB SAS disks
13 TB allocated 8 TB free 1.7 TB hot spare
Storage Pool [2] 1844 GB free
1 x 100 GB LUN
…
8 x 500 GB LUNs 2 x 500 GB LUNs
Oracle Database [5]
2.5 TB table & index data
Order Entry Workload
…
… 7 x 200 GB SLC SSDs
5100 GB allocated
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 24
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[4] The external redundancy policy is configured for all ASM disk groups. Other parameters keep the
default values.
[5] For details about the database configuration, see Table 3-4.
3.2 Verification Procedure
Step 1 Prepare the environment.
Deploy a network based on the solution architecture. Configure the OceanStor 5300 V3
converged storage system and create disk domains, storage pools, LUNs, and LUN groups as
instructed in the OceanStor V3 converged storage system user manual.
Pre-configure host operating systems as instructed in the Oracle 12c cluster deployment guide.
Set host parameters based on the following table. Install UltraPath as instructed in the
UltraPath user manual. Scan for hosts on the OceanStor 5300 V3 converged storage system.
Create host groups and mapping views.
Table 3-3 Host parameter settings
Configuration File Parameter Value
/etc/sysctl.conf vm.nr_hugepages 102400 (2 MB)
kernel.shmmax 214748364800 (bytes)
kernel.shmall 52428800 (4 KB)
/etc/security/limits.conf oracle soft nproc 16384
oracle hard nproc 65536
oracle soft nofile 16384
oracle hard nofile 65536
oracle soft memlock 209715200 (1 KB)
oracle hard memlock 209715200 (1 KB)
/sys/block/sd*/queue/scheduler deadline
/sys/block/sd*/queue/max_sectors_kb 1024
On each host, run the following commands to scan for LUNs, create UDEV policies, and
generate UDEV devices. (The commands only apply to Red Hat Enterprise Linux 6 and
UltraPath V100R008.)
upadm start hotscan
upadm show lun array=0 | grep -v LUNV | awk '/LUN Name/{printf "KERNEL==\"sd?*\",
BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u /dev/$name\", RESULT==\"3%s\",
NAME=\"huawei/%s\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"\n",$6,$9}' >
/etc/udev/rules.d/99-huawei-devices.rules
udevadm control --reload-rules
start_udev
ls -l /dev/huawei/
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 25
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Step 2 Deploy cluster and database software.
Deploy Oracle 12c cluster and database software as instructed in the Oracle 12c cluster and
database configuration guide. Use DBCA to create a container database. Set
db_create_file_dest to +DATA. Create five 128 MB log groups for each instance and enable
the archive mode. Place online logs and archive logs to the +LOG disk group. Set database
parameters based on the following table.
Table 3-4 Database parameter settings
Parameter Value
db_create_file_dest +DATA
db_block_size 8192
db_file_multiblock_read_count 128
sga_target 160 GB
pga_aggregate_target 40 GB
lock_sga TRUE
use_large_pages ONLY
db_files 512
process 1024
parallel_max_servers 256
fast_start_mttr_target 30
db_writer_processes 2
Step 3 Load data.
Use Huawei SwingBench Test Suite 1.0 for Oracle 12c to run data loading scripts that create
pluggable databases pdbt, pdbs, and pdbm and load test data at a scale of 250, 500, and 1000
for the three databases respectively (about 2 TB in total).
sh 0.load.sh pdbt 250 DATA
sh 0.load.sh pdbs 500 DATA
sh 0.load.sh pdbm 1000 DATA
The parameters of a data loading script (format: 0.load.sh pdb scale asmdg) are described as follows:
pdb: pluggable database name
scale: data scale, with about 1 GB per unit
asmdg: target ASM disk group without the plus symbol (+)
Step 4 Perform a warm-up test.
Run the following OLTP script to start a one-hour test. After that, manually start hotspot data
migration based on SmartTier.
sh 1.run.sh -p ora -c 2 -d pdbt,pdbs,pdbm -n 20,40,80 -i 10 -r 60 -w 10 -o xxx -s xxx
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 26
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
The parameters of an OLTP script (format: 1.run.sh -p NODE_PREFIX -c NODE_COUNT -d
PDB1,PDB2,PDB3... -n NumUsers1,NumUsers2,NumUsers3... -i INTERVAL -r
RUN_TIME_MINS -w WARMUP_MINS -o OS_PSWD -s SYSTEM_PSWD) are described as
follows:
-p NODE_PREFIX Oracle RAC node prefix, for example, RAC nodes are
'ora1,ora2,ora3', then set NODE_PREFIX to 'ora'
-c NODE_COUNT: Number of Oracle RAC nodes to run the test, if set to 2, then
swingbench will connect PDBs of 'ora1' and 'ora2'
-d PDB1,... PDBs to connect
-n NumUsers1,... Sessions to create for PDBs
-i INTERVAL Performance statistic & swingbench verbose print interval
seconds
-r RUN_TIME_MINS Run minutes, including warmup time
-w WARMUP_MINS Warm up minutes
-o OS_PSWD current OS user password
-s SYSTEM_PSWD password of dba user 'system'
In DeviceManager of OceanStor 5300 V3 converged storage system, manually migrate
hotspot data as follows:
1. Click the Provisioning icon on the right.
2. Click Resource Performance Tuning.
3. Click SmartTier.
4. Set Data Migration Speed to High.
5. Select the desired storage pool.
6. Choose More > Start.
Figure 3-3 Manually executing SmartTier hotspot data identification
Step 5 Start OLTP testing.
Run the following OLTP script to keep increasing the number of users until 20,000 transaction
IOPS is reached (the read latency is shorter than 10 ms.)
sh 1.run.sh -p ora -c 2 -d pdbt,pdbs,pdbm -n 35,70,140 -i 10 -r 20 -w 10 -o xxx -s xxx
---End
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 27
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
3.3 Verification Results
Table 3-5 Verification results
Database Amount of Data
Session Quantity
TPS IOPS I/O Latency
Disk Quantity
PDBT 389 GB 70 414
PDBS 730 GB 140 905
PDBM 1463 GB 280 1610
CDB 38 GB
Total 2620 GB 490 2929 28,877 8.83 ms 43 SAS
disks, 7 SSDs
Objective 2 TB 2000 20,000
Figure 3-4 OLTP test process
The y-axis indicates IOPS. The x-axis indicates the execution duration (expressed in minutes). riops
indicates read IOPS. wiops indicates write IOPS.
0
5000
10000
15000
20000
25000
30000
35000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
riops wiops
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 28
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
4 Small-Size OLTP Database Reference Architecture
4.1 Huawei Solution
4.1.1 Solution Architecture
Figure 4-1 Small-size OLTP database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
5500 V3 [2]
88 x 600 GB 10k rpm SAS disks12 x 200 GB SLC SSDs
4 x
12
Gb
it/s
SA
S
4 x 8 Gbit/s FC
3 x disk enclosures
Oracle 12c RAC
4 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Small-Size OLTP Database Solution4 TB[3] data, 40,000[4] transaction IOPS
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 29
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[2] The detailed configuration of the OceanStor 5500 V3 converged storage system is as follows: 48 GB
cache, 1 x controller enclosure (13 x 600 GB 10k rpm SAS disks, 12 x 200 GB SLC SSDs, 2 x 8 Gbit/s
four-port Fibre Channel I/O modules), 3 x disk enclosures (75 x 600 GB 10k rpm SAS disks), 1 x disk
domain containing all the 100 disks (46.1 TB capacity), 1 x storage pool (RAID 10 and 12 TB capacity
configured for the SAS tier, and RAID 5-9 and 1600 GB capacity configured for the SSD tier), 20 x 500
GB LUNs (16 LUNs used as the data area, and 4 LUNs used as the log area).
[3] The tested database contains 5 TB of table and index data.
[4] The tested transaction IOPS is 44,172, based on the Order Entry 2.0 order processing model.
4.1.2 Solution Configuration
Table 4-1 Hardware configuration
Solution Hardware Component Quantity
Small-Size
OLTP Database
Solution
Server: RH2288
V2
4
Intel® Xeon
® E5-2660 @ 2.20 GHz 2
Memory 256 GB
QLogic 8 Gbit/s dual-port Fibre
Channel HBA
1
Intel 10 Gbit/s dual-port Ethernet
HBA
1
Storage:
OceanStor 5500
V3
1
Cache 48 GB
2 U controller enclosure with 25
disk slots
1
2 U disk enclosure with 25 slots 3
600 GB 10k rpm SAS disk 88
200 GB SLC SSD 12
8 Gbit/s four-port Fibre Channel
I/O module
2
Fibre Channel
switch: SNS2224
2
Private network
switch: S6700
1
Table 4-2 Software configuration
Solution Hardware Software Quantity
Small-Size Data RH2288 V2 4
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 30
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Solution Hardware Software Quantity
Warehouse
Solution Operating system: Red Hat Enterprise
Linux 6.5
Multipathing software: UltraPath for
Linux 8.01.024
Database cluster: Oracle Grid 12.1.0.2
Database software: Oracle Database
12.1.0.2
Test tool: Huawei SwingBench Test
Suite 1.0 for Oracle 12c
OceanStor
5500 V3
1
SmartTier license
SmartThin license
SmartMotion license
SmartQoS license
Figure 4-2 Storage configuration
[1] The disk domain configuration is as follows: The hot spare policy of the SAS tier is High (default
policy), and that of the SSD tier is Low.
Disk
+DATA +LOG+GRID
...88 x 600 GB SAS disks
25.8 TB allocated 17.6 TB free 2.7 TB hot spare
3788 GB free
1 x 100 GB LUN
...
16 x 500 GB LUNs 4 x 500 GB LUNs
5 TB table & index data
Order Entry Workload
...
... 12 x 200 GB SLC SSDs
10,100 GB allocated
Disk Domain [1]
LUN [3]
ASM Disk Group [4]
Storage Pool [2]
Oracle Database [5]
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 31
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[2] The storage pool configuration is as follows: RAID 10 and 12 TB capacity are configured for the
SAS tier. RAID 5-9 and 1600 GB capacity are configured for the SSD tier. The SmartTier monitoring
period is from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day.
Other parameters keep the default values.
[3] The LUN configuration of the data area is as follows: The SmartTier policy is automatic migration.
LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
The LUN configuration of the log area is as follows: The read/write policy is reclamation. The priority is
high. LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
[4] The external redundancy policy is configured for all ASM disk groups. Other parameters keep the
default values.
[5] For details about the database configuration, see Table 3-4.
4.2 Verification Procedure
Step 1 Prepare the environment.
For details, see section 3.2 "Verification Procedure."
Step 2 Deploy cluster and database software.
For details, see section 3.2 "Verification Procedure."
Step 3 Load data.
Use Huawei SwingBench Test Suite 1.0 for Oracle 12c to run data loading scripts that create
pluggable databases pdbs, pdbm, and pdbl and load test data at a scale of 500, 1000, and
2000 for the three databases respectively (about 4 TB in total).
sh 0.load.sh pdbs 500 DATA
sh 0.load.sh pdbm 1000 DATA
sh 0.load.sh pdbl 2000 DATA
For details about the script parameter meanings, see section 3.2 "Verification Procedure."
Step 4 Perform a warm-up test.
For details, see section 3.2 "Verification Procedure."
Step 5 Start OLTP testing.
Run an OLTP script to keep increasing the number of users until 40,000 transaction IOPS is
reached (the read latency is shorter than 10 ms.)
sh 1.run.sh -p ora -c 4 -d pdbs,pdbm,pdbl -n 28,55,110 -i 10 -r 20 -w 10 -o xxx -s xxx
---End
4.3 Verification Results
Table 4-3 Verification results
Database Amount of Data
Session Quantity
TPS IOPS I/O Latency
Disk Quantity
PDBS 730 GB 112 665
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 32
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Database Amount of Data
Session Quantity
TPS IOPS I/O Latency
Disk Quantity
PDBM 1463 GB 220 1347
PDBL 2943 GB 440 2576
CDB 38 GB
Total 5140 GB 772 4588 44,172 7.84 ms 88 SAS disks,
12 SSDs
Objective 4 TB 4000 40,000
Figure 4-3 OLTP test process
The y-axis indicates IOPS. The x-axis indicates the execution duration (expressed in minutes). riops
indicates read IOPS. wiops indicates write IOPS.
0
10000
20000
30000
40000
50000
60000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 19 20
riops wiops
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 33
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
5 Medium-Size OLTP Database Reference Architecture
5.1 Huawei Solution
5.1.1 Solution Architecture
Figure 5-1 Medium-size OLTP database reference architecture
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
6 x
12 G
bit/s
SA
S
4 x 8 Gbit/s FC
6 x disk enclosures
Oracle 12c RAC
6 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Medium-Size OLTP Database Solution6 TB[3] data, 60,000[4] transaction IOPS
5600 V3 [2]
132 x 600 GB 10k rpm SAS disks18 x 200 GB SLC SSDs
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 34
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[2] The detailed configuration of the OceanStor 5600 V3 converged storage system is as follows: 64 GB
cache, 1 x controller enclosure (2 x 8 Gbit/s four-port Fibre Channel I/O modules, 2 x 12 Gbit/s
four-port SAS I/O modules), 6 x disk enclosures (132 x 600 GB 10k rpm SAS disks, 18 x 200 GB SLC
SSDs), 1 x disk domain containing all the 150 disks (69.1 TB capacity), 1 x storage pool (RAID 10 and
18 TB capacity configured for the SAS tier, and RAID 5-9 and 2400 GB capacity configured for the
SSD tier), 30 x 500 GB LUNs (24 LUNs used as the data area, and 6 LUNs used as the log area).
[3] The tested database contains 8 TB of table and index data.
[4] The tested transaction IOPS is 63,477, based on the Order Entry 2.0 order processing model.
5.1.2 Solution Configuration
Table 5-1 Hardware configuration
Solution Hardware Component Quantity
Medium-Size
Data
Warehouse
Solution
Server: RH2288
V2
6
Intel® Xeon
® E5-2660 @ 2.20 GHz 2
Memory 256 GB
QLogic 8 Gbit/s dual-port Fibre
Channel HBA
1
Intel 10 Gbit/s dual-port Ethernet
HBA
1
Storage:
OceanStor 5600
V3
1
Cache 64 GB
3 U controller enclosure 1
2 U disk enclosure with 25 slots 6
600 GB 10k rpm SAS disk 132
200 GB SLC SSD 18
8 Gbit/s four-port Fibre Channel
I/O module
2
12 Gbit/s four-port SAS I/O module 2
Fibre Channel
switch: SNS2224
2
Private network
switch: S6700
1
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 35
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Table 5-2 Software configuration
Solution Hardware Software Quantity
Medium-Size
Data
Warehouse
Solution
RH2288 V2 6
Operating system: Red Hat Enterprise
Linux 6.5
Multipathing software: UltraPath for
Linux 8.01.024
Database cluster: Oracle Grid 12.1.0.2
Database software: Oracle Database
12.1.0.2
Test tool: Huawei SwingBench Test
Suite 1.0 for Oracle 12c
OceanStor
5600 V3
1
SmartTier license
SmartThin license
SmartMotion license
SmartQoS license
Figure 5-2 Storage configuration
[1] The disk domain configuration is as follows: The hot spare policy of the SAS tier is High (default
policy), and that of the SSD tier is Low.
+DATA +LOG+GRID
...132 x 600 GB SAS disks
38.6 TB allocated 27.4 TB free 3 TB hot spare
5732 GB free
1 x 100 GB LUN
...
24 x 500 GB LUNs 6 x 500 GB LUNs
8 TB table & index data
Order Entry Workload
...
... 18 x 200 GB SLC SSDs
15,100 GB allocated
Disk Domain [1]
LUN [3]
ASM Disk Group [4]
Storage Pool [2]
Oracle Database [5]
Disk
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 36
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[2] The storage pool configuration is as follows: RAID 10 and 18 TB capacity are configured for the
SAS tier. RAID 5-9 and 2400 GB capacity are configured for the SSD tier. The SmartTier monitoring
period is from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day.
Other parameters keep the default values.
[3] The LUN configuration of the data area is as follows: The SmartTier policy is automatic migration.
LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
The LUN configuration of the log area is as follows: The read/write policy is reclamation. The priority is
high. LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
[4] The external redundancy policy is configured for all ASM disk groups. Other parameters keep the
default values.
[5] For details about the database configuration, see Table 3-4.
5.2 Verification Procedure
Step 1 Prepare the environment.
For details, see section 3.2 "Verification Procedure."
Step 2 Deploy cluster and database software.
For details, see section 3.2 "Verification Procedure."
Step 3 Load data.
Use Huawei SwingBench Test Suite 1.0 for Oracle 12c to run data loading scripts that create
pluggable databases pdbs, pdbm, and pdbxl and load test data at a scale of 500, 1000, and
4000 for the three databases respectively (about 6 TB in total).
sh 0.load.sh pdbs 500 DATA
sh 0.load.sh pdbm 1000 DATA
sh 0.load.sh pdbxl 4000 DATA
For details about the script parameter meanings, see section 3.2 "Verification Procedure."
Step 4 Perform a warm-up test.
For details, see section 3.2 "Verification Procedure."
Step 5 Start OLTP testing.
Run the following OLTP script to keep increasing the number of users until 60,000 transaction
IOPS is reached (the read latency is shorter than 10 ms.)
sh 1.run.sh -p ora -c 6 -d pdbs,pdbm,pdbxl -n 30,60,120 -i 10 -r 20 -w 10 -o xxx -s
xxx
---End
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 37
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
5.3 Verification Results
Table 5-3 Verification results
Database Amount of Data
Session Quantity
TPS IOPS I/O Latency
Disk Quantity
PDBS 752 GB 180 531
PDBM 1483 GB 360 1097
PDBXL 5759 GB 720 3870
CDB 41 GB
Total 8035 GB 1260 5498 63,477 9.66 ms 132 SAS disks,
18 SSDs
Objective 6 TB 60,000
Figure 5-3 OLTP test process
The y-axis indicates IOPS. The x-axis indicates the execution duration (expressed in minutes). riops
indicates read IOPS. wiops indicates write IOPS.
0
10000
20000
30000
40000
50000
60000
70000
80000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 17 18 19 20
riops wiops
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 38
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
6 Large-Size Transaction Database Reference Architecture
6.1 Huawei Solution
6.1.1 Solution Architecture
Figure 6-1 Large-size transaction database reference architecture
8 x
12 G
bit/s
SA
S
Oracle 12c RAC
8 x RH2288 V2 [1]
Cluster Private Interconnection
S6700 10GE switch
2 x SNS2224 FC switches
Large-Size OLTP Database Solution8 TB[3] data, 80,000[4] transaction IOPS
8 x disk enclosures
5800 V3 [2]
175 x 600 GB 10k rpm SAS disks25 x 200 GB SLC SSDs
4 x 8 Gbit/s FC
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 39
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x
QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.
[2] The detailed configuration of the OceanStor 5800 V3 converged storage system is as follows: 128
GB cache, 1 x controller enclosure (1 x 8 Gbit/s four-port Fibre Channel I/O module, 2 x 12 Gbit/s
four-port SAS I/O modules), 8 x disk enclosures (175 x 600 GB 10k rpm SAS disks, 25 x 200 GB SLC
SSDs), 1 x disk domain containing all the 200 disks (91.8 TB capacity), 1 x storage pool (RAID 10 and
24 TB capacity configured for the SAS tier, and RAID 5-9 and 3200 GB capacity configured for the
SSD tier), 40 x 500 GB LUNs (32 LUNs used as the data area, and 8 LUNs used as the log area).
[3] The tested database contains 10 TB of table and index data.
[4] The tested transaction IOPS is 86,429, based on the Order Entry 2.0 order processing model.
6.1.2 Solution Configuration
Table 6-1 Hardware configuration
Solution Hardware Component Quantity
Large-Size Data
Warehouse
Solution
Server: RH2288
V2
8
Intel® Xeon
® E5-2660 @ 2.20 GHz 2
Memory 256 GB
QLogic 8 Gbit/s FC dual-port HBA 1
Intel 10 Gbit/s dual-port Ethernet
HBA
1
Storage:
OceanStor 5800
V3
1
Cache 128 GB
3 U controller enclosure 1
2 U disk enclosure with 25 slots 8
600 GB 10k rpm SAS disk 175
200 GB SLC SSD 25
8 Gbit/s four-port Fibre Channel
I/O module
2
12 Gbit/s four-port SAS I/O module 2
Fibre Channel
switch: SNS2224
2
Private network
switch: S6700
1
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 40
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Table 6-2 Software configuration
Solution Hardware Software Quantity
Large-Size Data
Warehouse
Solution
RH2288 V2 8
Operating system: Red Hat Enterprise
Linux 6.5
Multipathing software: UltraPath for
Linux 8.01.024
Database cluster: Oracle Grid 12.1.0.2
Database software: Oracle Database
12.1.0.2
Test tool: Huawei SwingBench Test
Suite 1.0 for Oracle 12c
OceanStor
5800 V3
1
SmartTier license
SmartThin license
SmartMotion license
SmartQoS license
Figure 6-2 Storage configuration
+DATA +LOG+GRID
...175 x 600 GB SAS disks
51.5 TB allocated 36.6 TB free 3.7 TB hot spare
7676 GB free
1 x 100 GB LUN
...
32 x 500 GB LUNs 8 x 500 GB LUNs
10 TB table & index data
Order Entry Workload
...
... 25 x 200 GB SLC SSDs
20,100 GB allocated
Disk Domain [1]
LUN [3]
ASM Disk Group [4]
Storage Pool [2]
Oracle Database [5]
Disk
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 41
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
[1] The disk domain configuration is as follows: The hot spare policy of the SAS tier is High (default
policy), and that of the SSD tier is Low.
[2] The storage pool configuration is as follows: RAID 10 and 24 TB capacity are configured for the
SAS tier. RAID 5-9 and 3200 GB capacity are configured for the SSD tier. The SmartTier monitoring
period is from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day.
Other parameters keep the default values.
[3] The LUN configuration of the data area is as follows: The SmartTier policy is automatic migration.
LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
The LUN configuration of the log area is as follows: The read/write policy is reclamation. The priority is
high. LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.
[4] The external redundancy policy is configured for all ASM disk groups. Other parameters keep the
default values.
[5] For details about the database configuration, see Table 3-4.
6.2 Verification Procedure
Step 1 Prepare the environment.
For details, see section 3.2 "Verification Procedure."
Step 2 Deploy cluster and database software.
For details, see section 3.2 "Verification Procedure."
Step 3 Load data.
Use Huawei SwingBench Test Suite 1.0 for Oracle 12c to run data loading scripts that create
pluggable databases pdbm, pdbl, and pdbxland and load test data at a scale of 1000, 2000,
and 4000 for the three databases respectively (about 8 TB in total).
sh 0.load.sh pdbm 1000 DATA
sh 0.load.sh pdbl 2000 DATA
sh 0.load.sh pdbxl 4000 DATA
For details about the script parameter meanings, see section 3.2 "Verification Procedure."
Step 4 Perform a warm-up test.
For details, see section 3.2 "Verification Procedure."
Step 5 Start OLTP testing.
Run the following OLTP script to keep increasing the number of users until 80,000 transaction
IOPS is reached (the read latency is shorter than 10 ms.)
sh 1.run.sh -p ora -c 8 -d pdbm,pdbl,pdbxl -n 45,90,180 -i 10 -r 20 -w 10 -o xxx -s
xxx
---End
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 42
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
6.3 Verification Results
Table 6-3 Verification results
Database Amount of Data
Session Quantity
TPS IOPS I/O Latency
Disk Quantity
PDBM 1481 GB 360 1023
PDBL 2912 GB 720 2016
PDBXL 5753 GB 1440 3960
CDB 39 GB
Total 10,185 GB 1260 6999 86,429 9.54 ms 175 SAS disks,
25 SSDs
Objective 8 TB 80,000
Figure 6-3 OLTP test process
The y-axis indicates IOPS. The x-axis indicates the execution duration (expressed in minutes). riops
indicates read IOPS. wiops indicates write IOPS.
0
20000
40000
60000
80000
100000
120000
0 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20
riops wiops
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 43
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
7 Appendix
7.1 Reference Documents
For details about the best practices for deploying OLTP Oracle Database based on HUAWEI
OceanStor V3 converged storage systems, refer to the following document:
Planning and Configuring HUAWEI OceanStor V3 Converged Storage Systems to Maximize OLTP Oracle Database's Performance and Availability
7.2 Acronyms and Abbreviations
Table 7-1 Acronyms and abbreviations
Acronym and Abbreviation Full Name
ASM Automatic Storage Management
ARC Archive (Oracle log archive process)
AU Allocation Unit (of an ASM disk group)
BI Business Intelligence
CDB Oracle Container Database
CK Chunk
CKG Chunk Group
CKPT Checkpoint (Oracle checkpoint)
DBW DB Writer (Oracle disk flush process)
DG Disk Group
ETL Extract, Transform, Load (OLAP data processing phase)
FC Fibre Channel
FCoE Fibre Channel over Ethernet
GE Gigabytes Ethernet
HBA Host Bus Adapter
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 44
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Acronym and Abbreviation Full Name
HDD Hard Disk Drive
LUN Logical Unit Number (Huawei storage volume)
OCR Oracle Cluster Registry
OLAP Online Analytical Processing (oriented to complex
analytical query)
OLTP Online Transaction Processing (oriented to multi-user
online transactions)
PDB Oracle Pluggable Database
PGA Program Global Area (a memory region that contains
data and control information for a server process)
SAS Serial Access SCSI
SAN Storage Area Network
SGA System Global Area (a group of shared memory
structures that contain data and control information for
one Oracle database instance)
SSD Solid State Disk
NAS Network Attached Storage
NL-SAS Near-line SAS
RAC Real Application Clusters
VIP Virtual IP
Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 45
HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference
Architectures
Huawei Technologies Co., Ltd.
Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: http://www.huawei.com
Email: [email protected]
Copyright © Huawei Technologies Co., Ltd. 2015. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written
consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and the
customer. All or part of the products, services and features described in this document may not be within the
purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,
and recommendations in this document are provided "AS IS" without warranties, guarantees or
representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.