elements of san capacity planning mark friedman vp, storage technology [email protected] (239)...

Elements of SAN capacity planning

Mark FriedmanVP, Storage Technology

[email protected](239) 261-8945

04/19/23

Overview

How do we take what we know about storage processor performance and apply it to emerging SAN technology?

What is a SAN?

Planning for SANs:SAN performance characteristics

Original test results, May 2001: 4 x 550 MHz, 1 Gb FC More recent test results, May 2002: 2 x 2.266 GHz, 2 Gb

FC SPC-1 testing using 3 nodes with 2 x 3 GHz, 1 & 2 Gb FC

Backup and replication performance

Evolution Of Disk Storage Subsystems

See: Dr. Alexandre Brandwajn, “A study of cached RAID 5 I/O” CMG Proceedings, 1994.

Cached Disk

Strings & FarmsSpindles

Storage Processors

Write-thruCached

subsystems

04/19/23

Disk Subsystem Modeling framework:

ComponentsFront end interface bandwidthInternal bus bandwidthNumber of processors and their speedCache memoryDisk interfacesSCSI Disks

Internal segmented buffering Seek, Rotational Delay, Data transfer rate

Logical:physical disk mappingLinear mappingDisk stripingRAID mapping

04/19/23

What Is A SAN?

Storage Area Networks are designed to exploit Fibre Channel plumbing

Approaches to simplified networked storage:SAN appliances SAN Metadata Controllers (“out of band”)SAN storage managers (“in band”)

04/19/23

The Difference Between NAS and SANStorage Area Network (SAN) designed to exploit Fibre Channel plumbing require a new infrastructure.

Network Attached Storage (NAS) devices plug into the existing networking infrastructure.Networked file access protocols (NFS, SMB, CIFS)TCP/IP stack

Media Access: Ethernet, FDDI

Application: HTTP, RPC

Host-to-Host: TCP, UDP

Internet Protocol: IP

Packet Packet Packet Packet

04/19/23

The Difference Between NAS and SAN

NAS devices plug into existing TCP/IP networking support.

Performance considerations:1500 byte Ethernet MTU

TCP requires acknowledgement of each packet, limiting performance.

Application InterfacesRPC DCOM Winsock NetBIOS

Named Pipes

NetBT

Redirector Server

User ModeKernel

TCP UDP

IP ARPICMP IGMP IP Filtering IP Forwarding

Packet Scheduler

NDIS WrapperNDIS Miniport NIC Device Driver

TDI

NDIS

04/19/23

Source: Alteon Computers, 1999.

The Difference Between NAS and SAN

Performance considerations: e.g.,1.5 KB Ethernet MTU

Requires processing 80,000 Host interrupts/sec @ 1 Gb/sec

or Jumbo frames, which also requires installing a new infrastructure

Which is why Fibre Channel was designed the way it is!

04/19/23

The Holy Grail!

Storage Area NetworksUses low latency, high performance Fibre Channel switching technology (plumbing)

100 MB/sec Full duplex serial protocol over copper or fiber

Extended distance using fiberThree topologies:

Point-to-PointArbitrated Loop: 127 addresses, but can be bridged

Fabric: 16 MB addresses

04/19/23

The Holy Grail!

Storage Area NetworksFC delivers SCSI commands, but Fibre Channel exploitation requires new infrastructure and driver support

Objectives:Extended addressing of shared storage pools

Dynamic, hot-plugable interfacesRedundancy, replication & failover Security administrationStorage resource virtualization

04/19/23

Distributed Storage & Centralized Administration

Traditional tethered vs untethered SAN storage

Untethered storage can (hopefully) be pooled for centralized administration

Disk space pooling (virtualization)Currently, using LUN virtualizationIn the future, implementing dynamic virtual:real address mapping (e.g., the IBM Storage Tank)

Centralized back-upSAN LAN-free backup

04/19/23

FC is packet-oriented (designed for routing).FC pushes many networking functions into the hardware layer.e.g., Packet fragmentationRouting

Storage Area Networks

Upper Level Protocol

SCSI IPI-3 HIPPI IP Fc4

Framing Protocol/Flow ControlFc2

8B/10B Encode/Decode Fc1

100MB/s Physical Layer Fc0

Common Services Fc3

04/19/23

FC is designed to work with optical fiber and lasers consistent with Gigabit Ethernet hardware100 MB/sec interfaces200 MB/sec interfaces

This creates a new class of hardware that you must budget for: FC hubs and switches.


04/19/23

Performance characteristics of FC switches:Extremely low latency ( 1sec), except when cascaded switches require frame routing

Deliver dedicated 100 MB/sec point-to-point virtual circuit bandwidth

Measured 80 MB/sec effective data transfer rates per 100 MB/sec Port


04/19/23

Upper Level Protocol

SCSI IPI-3 HIPPI IP Fc4

Framing Protocol/Flow ControlFc2

8B/10B Encode/Decode Fc1

100MB/s Physical Layer Fc0

Common Services Fc3

When will IP and SCSI co-exist on the same network fabric?iSCSINishanOthers?


04/19/23


FC zoning is used to control access to resources (security)

Two approaches to SAN management:Management functions must migrate to the switch, storage processor, or….

OS must be extended to support FC topologies.

04/19/23

Approaches to building SANs

Fibre Channel-based Storage Area Networks (SANs)SAN appliancesSAN Metadata ControllersSAN Storage Managers

Architecture (and performance) considerations

04/19/23

Approaches to building SANs

Where does the logical device:physical device mapping run?Out-of-band: on the clientIn-band: inside the SAN appliance, transparent to the client

Many industry analysts have focused on this relatively unimportant distinction.

04/19/23

SAN appliances

Conventional storage processors withFibre Channel interfacesFibre Channel support

FC FabricZoningLUN virtualization

04/19/23

SAN Appliance Performance

Same as before, except faster Fibre Channel interfaces

Commodity processors, internal buses, disks, front-end and back-end interfaces

Proprietary storage processor architecture considerations

Internal Bus

Cache Memory

FC

In

terf

aces

FC

Dis

ks

Host

Inte

rfaces

Multiple Processors

04/19/23

SAN appliances

SAN and NAS convergence?Adding Fibre Channel interfaces and Fibre Channel support to a NAS box

SAN-NAS hybrids when SAN appliances are connected via TCP/IP.

Current Issues:Managing multiple boxesProprietary management platforms

04/19/23

SAN Metadata Controller

SAN ClientsSAN Clients

Pooled Storage Resources


SANMetadataController

1 3Token

Fibre Channel2

SAN clients acquire an access token from the Metadata Controller (out-of-band)

SAN clients then access disks directly using proprietary distributed file system

04/19/23

SAN Metadata Controller

Performance considerations:MDC latency (low access rate assumed)Additional latency to map client file system request to the distributed file system

Other administrative considerations:Requirement for client-side software is a burden!

04/19/23




Fibre Channel

StorageDomainServers

SAN Storage Manager

Requires all access to pooled disks through the SAN Storage Manager (in-band)!

04/19/23




Fibre Channel

Storage

Domain

Servers

SAN Storage Manager

SAN Storage Manager adds latency to every I/O request

How much latency is involved?

Can this latency be reduced using traditional disk caching strategies?

04/19/23

Architecture of a Storage Domain ServerRuns on an ordinary Win2K Intel server

The SDS intercepts SAN I/O requests, impersonating a SCSI disk

Leverages:Native Device driversDisk managementSecurityNative CIFS support

Fibre Channel HBA Driver

SCSI miniport Driver

Disk Driver

Diskperf (measurement)

Fault Tolerance (Optional)

Data Cache

Fault Tolerance

Initiator/Target Emulation

FC Adaptor Polling Threads

Security

Natives W2K I/O Manager

SANsymphony Storage Domain Server

Client I/O

04/19/23

Sizing the SAN Storage Manager serverIn-band latency is a function of (Intel server) front-end bandwidth:Processor speedNumber of processorsPCI bus bandwidthNumber of HBAs

and performance of the back-end Disk configuration

04/19/23

SAN Storage Manager

Can SAN Storage Manager in-band latency be reduced using traditional disk caching strategies?

Read hitsRead misses

Disk I/O + (2 * data transfer)

Fast Writes to cache (with mirrored caches) 2 * data transfer Write performance ultimately determined by the

disk configuration

04/19/23

SAN Storage Manager

Read hits (16 KB block):Timings from an FC hardware monitor1Gbit/s Interfaces

No bus arbitration delays!

140sec 27sec

StatusFrame

SCSI ReadCommandLength =

400016x1024 Byte Data Frames

04/19/23

SCSI Command Write Setup Data Frames SCSI Status

Fibre Channel Latency (16KB Blocks)

Read vs. Write hits (16 KB block)

04/19/23

SCSI Command Write Setup Data Frames SCSI Status

How is time being spent inside the server?

PCI bus?Host Bus adaptor?Device polling?Software stack?

Decomposing SAN in-band Latency

04/19/23

Memory Bus

64b

it/3

3M

Hz

PC

I

32b

it/3

3M

Hz

PC

I

32b

it/3

3M

Hz

PC

I

4x550MHzXEON

Processors

4-way 550 MHz PCMaximum of three FC interface polling threads

3 PCI buses (528MB/s Total)

1, 4, or 8 QLogic 2200 HBAs

Benchmark Configuration

04/19/23

How is time being spent inside the SDS?PCI bus?Host Bus adaptor?Device polling:

1 CPU is capable of 375,000 unproductive polls/sec 2.66secs per poll

Software stack:3 CPUs are capable of fielding 40,000 Read I/Os per second from cache

73secs per 512-byte I/O


04/19/23

SDS FC Interface Data Transfer

SANsymphony in-band Latency (16KB Blocks)


04/19/23

Impact Of New Technologies

Front-end bandwidth:Different speed ProcessorsDifferent number of processorsFaster PCI BusFaster HBAs

e.g. Next Generation Server2GHz GHz Processors (4x Benchmark System)200MB/sec FC interfaces (2x Benchmark System)4x800MB/s PCI bus (6x Benchmark System)

...

04/19/23


2GHz CPU & New HBAs

2GHz CPU, New HBAs, 2Gbit Switching

1 year ago

04/19/23


May 2001 May 2002

4 x 550 MHz,1 Gb FC, PCI-2

2 x 2266 Mhz,2 GB FC, PCI-X

Unproductive (idle) poll 2.66 secs 0.62 secs

CPU time Cache hit (read) 75 secs 25 secs

Throughput (16 KB) 270 MB/sec 952 MB/sec

I/Os per second 40,000 180,000

04/19/23

Sizing the SAN Storage Manager

ScalabilityProcessor speedNumber of processorsPCI bus bandwidth

32bit/33MHz 132MB/sec 64bit/33MHz 267MB/sec 64bit/66MHz 528MB/sec 64bit/100MHz 800MB/s (PCI-X)

NGIO???Number of HBAs

200 MB/sec FC interfaces feature faster internal processors

04/19/23

Sizing the SAN Storage Manager

Entry level system: Dual Processor, single PCI bus, 1 GB RAM

Mid-level departmental system: Dual Processor, dual PCI bus, 2 GB RAM

Enterprise-class system: Quad Processor, triple PCI bus, 4 GB RAM

04/19/23

0

10,000

20,000

30,000

40,000

50,000

0 2 4 6 8 10FC HBA s

Max

IO

s p

er s

eco

nd

(@

512

by

tes)

0

50

100

150

200

250

300

Max

MB

/sec

(@

16 K

B)

Max Read I/ Os Max W rite I/ Os

Max Read Throughput Max W rite Throughput

SAN Storage Manager PC scalability

04/19/23

0

10,000

20,000

30,000

40,000

50,000

0 2 4 6 8 10FC HBA s

Max

IO

s p

er s

eco

nd

(@

512

by

tes)

0

50

100

150

200

250

300

Max

MB

/sec

(@

16 K

B)

Max Read I/ Os Max W rite I/ Os

Max Read Throughput Max W rite Throughput

Entry level

Departmental SAN

Enterprise class


04/19/23

May 2002


SAN Appliance performance

0

200

400

600

800

1000

1200

100 1000 10000 100000 1000000 10000000

Block size

Th

rup

ut

(MB

/se

c)

0

40000

80000

120000

160000

200000

240000

I/O

s p

er

se

co

nd

Thruput (MB/sec) IOs per second

DataCore SPC-1 Results (August 2003)

www.storageperformance.org/Results/SPC-1/DataCore_2003-08-11_SANsymphony/

SPC-1 Submission Identifier: A00015

Tested Storage Configuration (TSC) Name: DataCore SANsymphony Network Edition

Metric Reported Results

SPC-1 IOPs 50,003.55

SPC-1 Price-Performance $6.11/SPC-1 IOPS™

Total ASU Capacity 1,407GB

Data Protection Level Mirroring

SPC-1 LRT 1.68 ms

Total TSC Price (including three-year maintenance)

$305,608

Fujitsu Softek SPC-1 Results (August 2003)

http://www.storageperformance.org/Results/SPC1/Fujitsu_2003_08_11_ETERNUS3000-M600M/2003-08-11_Fujitsu_ETERNUS3000-

M600M_SPC1-FDR.pdfSPC-1 Submission Identifier: A00016

Tested Storage Configuration (TSC) Name: Fujitsu Storage Systems ETERNUS 3000 Model 600M

Metric Reported Results

SPC-1 IOPs 64,249.77

SPC-1 Price-Performance $32.72/SPC-1 IOPS™

Total ASU Capacity 15,609GB

Data Protection Level Mirroring

SPC-1 LRT 2.31 ms

Total TSC Price (including three-year maintenance)

$2,102,147

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

$0$10$20$30$40

Price per SPC-1 IOPS

SP

C -

1 IO

PS

DataCore 3-nodeSun 63203PAR 8-nodeFujitsu ETERNUSIBM ESS 800HP EVA 2C12DLSI E4600

DataCore SANSymphony Cost/Performance

50,003SPC-1 IOPS™

50,003SPC-1 IOPS™

$6.11 perSPC-1 IOPS™

$6.11 perSPC-1 IOPS™

04/19/23

SANsymphony Performance

ConclusionsFC switches provide virtually unlimited bandwidth with exceptionally low latency so long as you do not cascade switches

General purpose Intel PCs are a great source of inexpensive MIPS.

In-band SAN management is not a CPU-bound process.

PCI bandwidth is the most significant bottleneck in the Intel architecture.

FC Interface cards speeds and feeds are also very significant

04/19/23

SAN Disk Subsystem Modeling framework:

ComponentsFC front end interface bandwidthInternal PCI bus bandwidthNumber of Intel processors and their speedCache memoryFibre Channel Disk interfacesSCSI Disks

Internal segmented buffering Seek, Rotational Delay, Data transfer rate

Logical:physical disk mappingLinear mappingDisk stripingRAID mapping

04/19/23

SAN Storage Manager – Next Steps

Cacheability of Unix and NT workloadsDomino, MS ExchangeOracle, SQL Server, Apache, IIS

Given mirrored writes, what is the effect of different physical disk configurations?JBODRAID 0 disk stripingRAID 5 write penalty

Asynchronous disk mirroring over long distances

Backup and Replication (snapshot)

04/19/23

??Questions

www.datacore.com

elements of san capacity planning mark friedman vp, storage technology [email protected] (239)...

Documents