ip storage tutorial presented 17 october 2001 by marc staimer, president & cds – dragon slayer...

72
IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel John Hufferd, Sr. Technical Staff – IBM SSD Joe Gervais, Director Product Marketing – Alacritech

Post on 20-Dec-2015

218 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

IP Storage Tutorial

Presented 17 October 2001 byMarc Staimer, President & CDS – Dragon Slayer Consulting

Ahmad Zamer, Sr. Product Line Marketing – Intel

John Hufferd, Sr. Technical Staff – IBM SSD

Joe Gervais, Director Product Marketing – Alacritech

Page 2: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

Tutorial Introduction

Marc Staimer, CDS – Dragon Slayer [email protected]

Page 3: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 3

The Purpose of this Tutorial

IP Storage as “block” vs. “file” storage NAS will be discussed peripherally

To provide details about IP Storage To provide factual information To clarify issues To facilitate understanding Key point

This is will be pragmatic education not cheerleading

Page 4: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

IP Networked Storage

iSCSI – New Possibilities

Ahmad Zamer

[email protected]

October 2001

Page 5: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 5

Overview

Introduction Benefits of IP Storage IP Storage technologies iSCSI Conclusions

Page 6: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 6

Introduction

“Ethernet wins. Again. In time… Ethernet will eventually triumph over all other storage networking technologies, including Fibre Channel”

Source: March 2001 Forrester Research

“If we were starting with a clean piece of paper … we would probably use gigabit Ethernet and IP” Source: Bill Miller CTO StorageNetworks, Industry Standard

“... 76% of senior IT executives believe IP will make it easier to implement large-scale storage networks”

Source: Enterprise Storage Group 9/11/2000

“75% perceive iSCSI as the IP storage standard”Source: Marc Staimer , Dragon Slayer Consulting – May 2001

Page 7: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 7

Network Storage Models

Direct Attached Storage Network Attached Storage Storage Area Network

•High Cost of Ownership•In-flexible

•Transmission optimized for file transactions•Storage traffic travels across the LAN

•Transmission optimized for database transactions•Separate LAN and SAN•Increases Data availability•Flexible and scalable

Page 8: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 8

Embedded

Direct attached

Type text

Type text

Type text Type text

Type text

Type tex t

Type text

Type text

LAN attachedNAS (TCP/IP)

SAN Fabric

or Loop

SAN AttachedFC

Shared

Networked StorageNetworked StorageStorage technology is moving away from dedicated storage to networked storage

Inch

feet

Miles

Internet

InfiniBandSwitch

CPUs

Storage

InfiniBandInfiniBand

TCP/IP Fabric

Servers

Storage

IP based SANIP based SAN Capitalize on existing IP infrastructureConvergence of SAN and NAS TechnologyAny to any connectivity, vendor independentHW TCP/IP Engines on HBAs

Replacement of CPU I/O Bus architectureFast access to shared memory and I/OWill evolve to a data center storage connectivity solution

CentralizedCentralized

DistributedDistributed

Moving from Dedicated to Networked Storage

Page 9: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 9

Benefits of IP Storage

Brings the SAN concept to Ethernet networks Lower total cost of ownership Creates a single integrated network Makes remote data replication possible Improves enterprise networks management Provides higher degree of interoperability

Page 10: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 10

Advantages of IP Storage

Storage access over distance Transparent to Applications Leverage Benefits of IP

IT Skills Ethernet & SCSI Infrastructure Network Management R&D Investment

Universal Access to Storage

GE

Storage Router

IP Network

FC or SCSI

iSCSI

Storage appears local to servers

Page 11: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 11

Key Business Trends Favor IP Storage

Network Performance

Trained Staff Available Total Cost of Ownership

Overall System Cost

1Gbps

0.85Gbps

10Gbps

40Gbps

100Gbps

1.7Gbps

10Gbps

FC SwitchesFC Switches

2000 2001 2002 2003

IP StorageSwitches

IP StorageSwitches

FC SwitchesFC Switches

2000 2001 2002 2003

IP StorageSwitches

IP StorageSwitches

FC SwitchesFC Switches

2000 2001 2002 2003

IP StorageSwitches

IP StorageSwitches

FC SwitchesFC Switches

2000 2001 2002 2003

IP StorageSwitches

IP StorageSwitches

Page 12: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 12

IP Storage Standards

IETF IP Storage (IPS) Working Group iSCSI FCIP iFCP iSNS

Storage Networking Industry Association (SNIA) SNIA IP Storage Forum

Storage Networking Industry Association

Page 13: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

IP Storage Technologies

Page 14: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 14

What are the technologies? (iSCSI, iFCP, FCIP)

iSCSI iSCSI is a TCP/IP-based protocol for establishing and managing

connections between IP-based storage devices, hosts and clients

FCIP FCIP is a TCP/IP-based tunneling protocol for connecting

geographically distributed Fibre Channel SANs transparently to both FC and IP

iFCP iFCP is a TCP/IP-based protocol for interconnecting Fibre

Channel storage devices or Fibre Channel SANs using an IP infrastructure in place of Fibre Channel switching and routing elements

Page 15: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 15

IP Storage: iSCSI, FCIP, iFCP

iSCSI iSCSI/IPInternetProtocol

FCIPFibre

ChannelFibre

Channel

iFCPFibre

ChannelInternetProtocol

EndEndDevicesDevices

FabricFabricServicesServices**

* Fabric Services include routing, device discovery, management, authentication, inter-switch communication

Page 16: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 16

iSCSI, iFCP and FCIP Protocol Stacks

IP IP IP

TCP TCP

New Serial SCSI FCP FC-4

TCP

FC Lower Layers

FCP FC-4

iSCSI iFCP FCIP

Standard SCSI Command Set

Operating System

Applications

Page 17: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

iFCP

Page 18: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 18

iFCP

iFCP is a gateway-to-gateway protocol for implementing a fibre channel fabric over a TCP/IP transport

Traffic between fibre channel devices is routed and switched by TCP/IP network

The iFCP layer maps Fibre Channel frames to a predetermined TCP connection for transport

FC messaging and routing services are terminated at the gateways so the fabrics are not merged to one another

Dynamically creates IP tunnels for FC frames

//EthernetHeader

CRCIP

ChecksumChecksum

TCP iFCP SCSI Data … FCP

Page 19: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 19

IP Network

IP Services at individual device levelIETF Standards for Routing, Naming,Security, QoS, CoS, Discovery (iSNS)

iFCPGateway

iFCPGateway

iFCPGateway

FCServer

FCServer

FC TapeLibrary

FCServer

FCServer

FC TapeLibrary

Device-to-DeviceSession

Device-to-DeviceSession

iFCP Approach

FCJBOD

FCJBOD

iSNS Server

iSNS Server

iFCPGateway

iFCPGateway

iFCPGateway

iFCPGateway

iFCPGateway

iFCP provides F port to F port connectivity only

Page 20: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

FCIP

Page 21: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 21

FCIP

FCIP encapsulates FC frames within TCP/IP, allowing islands of FC SANs to be interconnected over an IP-based network

TCP/IP is used as the underlying transport to provide congestion control and in-order delivery FC Frames

All classes of FC frames are treated the same as datagrams End-station addressing, address resolution, message routing, and

other elements of the FC network architecture remain unchanged IP introduced exclusively as a transport protocol for an inter-network

bridging function IP is unaware of the Fibre Channel Payload and the FC fabric is

unaware of IP

EthernetHeader

CRCIP

ChecksumChecksum

TCP FCIP SCSI Data …

//

FCP

Page 22: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 22

FCIP Approach—IP Tunneling

IP NetworkTunnel SessionTunnel Session

IP ServicesAvailable at Aggregated

FC SAN Level

FC TapeLibrary

FC TapeLibrary

FC Server

FC Server FC

ServerFC

JBOD

FCJBOD

FC Switch

FC SwitchFC Switch

FC SwitchFC Switch

FC Switch

FC Switch

Fibre Channel

SAN

Fibre Channel

SAN

FCIPTunnel

FCIPTunnel

FC Server

FCIP provides E port to E port connectivity

Page 23: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

iSCSI

Page 24: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 24

iSCSI

iSCSI is a SCSI transport protocol for mapping of block-oriented storage data over TCP/IP networks

The iSCSI protocol enables universal access to storage devices and Storage Area Networks (SANs) over standard TCP/IP networks

EthernetHeader CRCIP

ChecksumChecksum

TCP iSCSI SCSI Data…

//

Page 25: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 25

iSCSI, iFCP, FCiP

EthernetHeader CRCIP

ChecksumChecksum

TCP iSCSI SCSI Data…

//

EthernetHeader

CRCIP

ChecksumChecksum

TCP iFCP SCSI Data … FCP

EthernetHeader CRCIP

ChecksumChecksum

TCP FCIP SCSI Data …

//

FCP

Page 26: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 26

iSCSI – Cont.

iSCSI (Internet SCSI) specifies a way to “encapsulate” SCSI commands in a TCP/IP network connection:

SCSI commands and dataiSCSI Header

TCPHeader

IP Header

Explains how to extractSCSI commands and data

Provides information necessary toguarantee delivery

Contain “routing” informationSo that the message can find itsWay through the network

Page 27: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 27

In Combination

with NAS

iSCSI DeploymentsiSCSI Deployments

SAN

IPNetwork

SCSI Protocol

Multi Function Gateway

Block and File IO

NAS

iSCSIiSCSI

NAS

IndependentiSCSI

DeploymentApplication Host

SCSI Protocols

IPNetwork

Extending the SAN

SCSI Protocol

SAN

IPNetwork iSCSI Gateway

SCSI Protocol

Same HW Configurations as NASWorkgroup, Departmental, & Enterprise

(Appliances and Gateways)GAs throughout 2001 & 2002

iSCSI Deployment

Page 28: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 28

iSCSI Implementations

iSCSI Gateway FC

Switch Disk

Native iSCSI DeviceiSCSI Client

iSCSI Server

IP Network

Page 29: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 29

Storage Consolidation

Server and LAN bottlenecks Single points of failure Poor scalability (management overhead, resource

inefficiencies)

Tape Drives => Tape Library Departmental => Application-centric disc

arrays

LAN

NTServers

Tape Drive

RAID

Tape Drive

RAID

Tape Drive

RAID

NTServers

RAID(Email)

TapeLibrary

Mission-Critical RAID(Oracle, ERP DB)

SAN

Switch Switch

SwitchSwitch

Page 30: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 30

iSCSI Architecture

Overview Architectural Model Features Beyond // SCSI Issues Beyond // SCSI

Page 31: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 31

iSCSI - Layered Model

Replaces shared bus with switched fabric Transparently encapsulates SCSI CDBs Unlimited target and initiator connectivity

SCSI Application

iSCSI ProtocolServices

SCSI DeviceServer

iSCSI ProtocolServices

Initiator I/O System Target I/O System

SCSI CDB

iSCSI ProtocolLayer

TCP/IP

EthernetData link +Physical

Data link +Physical

SCSIApplication

Layer

iSCSI PDU

TCP segmentsin IP

datagrams

EthernetFrame

SCSI ApplicationProtocol

iSCSI Protocol

Ethernet

Protocol ServiceInterface

iSCSI TransportInterface

TCP/IPTCP/IPTCP/IP

TCP/IPTCP/IP

TCP/IP TCP/IP Protocol

iSCSI session

Page 32: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 32

iSCSI SessionsiSCSI Sessions

Session between initiator and targetOne or more TCP connections per sessionLogin phase begins each connection

Deliver SCSI commands in order Recover from lost connections

iSCSI DeviceiSCSI Host

iSCSI Initiator iSCSI TargetiSCSI SessioniSCSI Session

TCP Connection

TCP Connection

iSCSI Target

iSCSI SessioniSCSI Session

TCP Connection

Page 33: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 33

IPNetwork

iSCSI Encapsulation

LUNs

Fibre Channel SAN

Data Servers

End Users

SCSI Target

iSCSI InitiatorSCSI Initiator

iSCSI TargetiSCSI Target

EthernetHeader

IP

TCP

DATACRC

SCSIiSCSI

FC Header DATA

CRC

SCSI

ExternalNetwork

EthernetHeader

IP

TCP

DATACRC

Page 34: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 34

IPNetwork

iSCSI Packet Order

LUNs

Fibre Channel SAN

Data Servers

SCSI Target

iSCSI InitiatorSCSI Initiator

iSCSI Target 1 3 2

1 2 3

1 2 3

Page 35: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 35

iSCSI Packet

EthernetHeader

CRCIP

ChecksumChecksum

TCP iSCSI SCSI Data…

//

Page 36: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 36

iSCSI Packet

DestinationAddress

SourceAddress

Type IP TCP Data

46–1500 bytes

8 6 6 2

FCS

4 Octet

Preamble

TCP Header

Sourced Port

iSCSI Encapsulated

Opcode Opcode Specific Fields

Length of Data (after 40Byte header)

LUN or Opcode-specific fields

Initiator Task Tag

Opcode Specific Fields

Data Field …

Destination Port

Well-known Ports:

21 FTP 23 Telnet 25 SMTP 80 http5003 iSCSI

Sequence Number

Acknowledgment Number

Window

Checksum

OffsetReservedU A P R S F

Options and Padding

Urgent Pointer

Page 37: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 37

iSCSI Commands

SCSI Commands Command phase Optional data phase Response phase iSCSI Commands

Binds command phase with associated data into iSCSI Protocol Data Unit (PDU)

Page 38: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 38

iSCSI Architecture Features Beyond // SCSI

Sessions Comprises one or more TCP connections used for fail

over and/or link aggregation

Device sharing Any host on the network can potentially use the same

iSCSI device

Device scalability Hosts can connect to an effectively limitless number of

iSCSI devices

Page 39: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 39

iSCSI Architecture Issues Beyond // SCSI

Naming, addressing and discovering Security & Data Integrity Ordering and numbering Error handling/recovery Networking Overhead

Page 40: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 40

iSCSI Architecture IssuesNaming, Addressing & Discovery

// SCSI uses a simple NAD scheme: Devices discovered by polling the bus Devices given unique id between 0 and 15

iSCSI requires: Internet addressing Location independent naming

operation beyond firewallsmultiple addresses to one targetmultiple targets behind one address3rd party commands

Scalable discovery (poll the Internet??)

Page 41: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 41

1) Host driver requests available iSCSI targets from the SCSI router

2) SCSI router sends available iSCSI target names to host

3) Host logs into iSCSI targets that were received

4) SCSI router accepts the login and sends target identifiers to Host (numbers)

5) Host queries targets for device information

6) Targets respond with device information

7) Host creates table of internal devices (/dev/…)

iSCSI Storage Device Discovery Process

Page 42: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 42

Send Targets

0X03 Command—Login

iSCSI Sequence

TCP port 5003

TargetInitiator Single TCP Session

Establish normal TCP SessionTCP

iSCSI DriveriSCSI Driver

0X43 Login Response—Reject Login Status 1

In text area, list of assessable target names. Keeps TCP session up.

0X03 Command—LoginList of Target names sent

This device

has already

initialized onto the

Fibre Channel

This device

has already

initialized onto the

Fibre Channel

0X43 Login Response

Response with target drive mapping

Page 43: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 43

iSCSI Architecture Issues: Security Levels

0: None – ok in controlled environments 1: Initiator and target authentication

Prevents unauthorized access

2: Digests for header and data integrity Prevents against man-in-middle, insertion,

modification and deletion

3: Encryption (IPSEC) Prevents against eavesdropping

Page 44: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 44

iSCSI Architecture Issues Ordering & Numbering

Unlike // SCSI, iSCSI PDUs may Arrive out of order (by taking different routes) Not arrive at all

iSCSI requires Command numbering

Ordered delivery over multiple connections

Status numberingDetection of a failed connections

Data sequencingDetection of missing data PDUs

Page 45: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 45

iSCSI Architecture Issues Error Handling & Recovery

// SCSI errors incur costly recovery: Aborted commands; target, bus and host resets OK, because bus errors are infrequent

iSCSI errors will be more frequent Link failures TCP failures Bad “middle box” (firewall, router) Does the Internet have a “reset” option??

Page 46: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 46

iSCSI Architecture Issues Networking Overhead

Software iSCSI can achieve near GbE wire speed – but at 100% CPU

Traditional TCP stacks are expensive multiple memory copies too many interrupts checksums calculations

We needs TCP offload engines (TOE)

Page 47: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 47

iSCSI - TCP Offload

Ethernet frame requires additional CPU processing Headers must be stripped Packets ordered Data copied into memory buffers CRC checked

EthernetHeader

IP TCP iSCSI SCSI Data CRC

Page 48: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 48

iSCSI Architecture Issues Networking

TOE The challenge rests on the TOE vendor

Interrupt host on command boundariesOffer zero-copy from NIC to appEliminate TCP reassembly buffer

Provides true zero-copy Requires RDMA or synchronization

Proposed IETF solutions for framingWARP - an RDMA mechanismMarkers – a synchronization mechanism

Page 49: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 49

What’s Next for iSCSI

CRC SLP (Service Location Protocol) Authentication Encryption

Page 50: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

Conclusions

Page 51: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 51

Conclusions

IP-based storage will proliferate Benefits are strong Significant players Clear need Standards will be established Work with industry leaders

Page 52: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

Backup

Page 53: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 53

iSNS

iSNS (Internet Storage Name Server) Provides registration and discovery of SCSI

devices and Fibre Channel-based In IP-based storage like iSCSI end devices

registered with iSNS In iFCP, Fibre Channel-based storage end

devices register with iSNS by a iFCP gateway

Page 54: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 54

iSNS Operation

Server_1

Local iFCP Portal

RemoteiFCP portalIP

Network

IP address10.1.2.3

IP address10.1.2.4

N_port ID#24

N_port ID#24

Server_2

Problem: Two identical N_port IDs

Solution: Create new ID (based on IP address + N_port ID) = 2422

iSNS server

FC network 1 FC network 2

Page 55: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 55

Tracing an iSCSI Block I/O

En

capsulation

De-

enca

psul

atio

n

Server iSCSI Appliance

File I/O requests

Database Application Application

Operating System

Database System File System

Raw Partition Manager Volume Manager

SCSI Device Driver

iSCSI Device Driver Layer

TCP/IPP stack

Network Interface CardNetwork Interface Card

TCP/IPP stack

iSCSI Device Driver Layer

SCSI Device Driver

RAID Host Bus Adapter

Storage I/O Bus

iSCSI Appliance Storage21

Device specific requests to TCP/IP network

Block I/O / data / storage location

Page 56: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 56

Challenge 1 - TCP Overhead

Consider a SCSI WRITE command. How many times do you think the data is copied before eventually reaching the target HBA?

Application –copy-> Buffer Cache –copy-> TCP/IP –DMA-> Ether (2 copies 1 DMA) Ether –DMA-> Ring Buffer –copy-> TCP/IP –copy-> Bridge –DMA-> HBA (2 copies 2 DMA)

Buffer Cache

SCSI Subsystem

iSCSI Host Driver

Application

File System

Linux Host System

Ether

TCP/IP

Ethernet Driver

Bridging Software

iSCSI Target Driver

TCP/IP

Ethernet Driver

Block Device Driver

Linux Target System

Ether HBA

12 3

4

Page 57: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 57

TCP Overhead (2)

TCP Processing Every TCP connection that is part of an iSCSI session has

processing overhead potentialConnection setup / teardownTCP state machine:

Acknowledge, Timeout, Retransmission Window management Congestion Control

TCP segmentation IP fragmentationChecksum calculations

Partial or Complete TCP Offload mechanisms are assumed to be required to make iSCSI performance comparable to FC

Page 58: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 58

Challenge #2 – Framing

Message Boundaries (The Framing - HW-Issue) iSCSI messages have no alignment relationship with TCP

segments And TCP does not have a “built in mechanism” for signaling

message boundaries. IETF considered leverage the urgent pointer for some time

So how can an iSCSI adapter determine where a message begins and ends??

By reading the length field in the iSCSI header Determines where in byte stream current message ends and next

begins NIC must stay “in sync” with beginning of byte stream Works well in a perfect world (Maybe a SAN or LAN ????)

In a MAN/WAN we have issues IP Frags leading to out-of-order packet delivery and/or packet loss Any “middle box” may fragment an IP packet until, sending each

along potentially different routes

Page 59: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 59

Framing (2)

Message Boundaries Continued THE SCENARIO:

An iSCSI header is not received when expected because the TCP segment that it was part of was delivered out of order

THE ISSUE: The receiver does not know where to put the trailing data packets

until the packet with the header arrives The different options?

Drop all packets until the header arrives They will be retransmitted

Buffer packets until the header arrives. Then “re-assemble.” On a 1Gbit WAN link,16MB of buffer memory is required per TCP

connection On a 10 Gbit WAN link, 125MB of buffer memory required per TCP

connection

Page 60: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 60

Framing (3)

Message Boundaries Continued THE BAD NEWS:

Dropping packets greatly impacts performance and significantly increases network congestion

Local buffering is expensive and NIC logic is complex

Page 61: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 61

Into – SAN View

Hosts

Targets

Infrastructure

Primary StoragePrimary Storage

Secondary StorageSecondary Storage

Storage Management & Apps

Page 62: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 62

SAN Components

Server Platforms: Fibre Channel Host Bus Adapters IP Storage NICs (SNICs) SAN Software

Storage Platforms: RAID subsystems JBOD Tape subsystems

SAN Interconnect: Fibre Channel hubs and switches IP Storage switches SAN-to-SCSI bridges MAN and WAN gateways

Page 63: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 63

SAN, NAS, iSCSI Comparison

DAS

SCSI

Computer System

SCSI Bus Adapter

SCSI Device Driver

Volume Manager

File System

Application

SAN

SAN

FC

Fibre Channel HBA

SCSI Device Driver

Volume Manager

File System

Application

iSCSI Appliance

IP

File System

Application

SCSI Device DriveriSCSI Driver

TCP/IP stack

NIC

Volume Manager

NIC

TCP/IP stack

iSCSI layerBus Adapter

iSCSI Gateway

IP

FC switch

File System

Application

SCSI Device DriveriSCSI Driver

TCP/IP stack

NIC

Volume Manager

NIC

TCP/IP stack

iSCSI layerBus Adapter

NAS

IP

NIC

TCP/IP stack

I/O Redirector

File System

Application

NFS/CIFS

NIC

TCP/IP stack

File SystemDevice driver

File I/O

Block I/O

Block I/O

Page 64: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 64

Functional Placement / Processing Cycles

SAN NAS iSCSI

FC NetworkSCSI ProtocolsSwitched FCS1 or 2 Gbps

Storage

Application

IP(TCP / UDP)

NFS/CIFS File System

NFS/CIFS Client

IP NetworkFile Protocols(CIFS, NFS...)Switched enet1 Gbps

IP NetworkiSCSI ProtocolsSwitched enet1 Gbps

File System

Sto

rag

e M

gt

Sto

rag

e M

gt

File System

Sto

rag

e M

gt

IP(TCP / UDP)

Storage Mgt

FC Protocol

Storage

Storage Mgt

IP(TCP/UDP)

Storage

Contrasting Storage Network Technologies Contrasting Storage Network Technologies

ApplicationServer

Network

StorageServer

Application

FC Protocol

Application

IP(TCP / UDP)

Page 65: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

65 17 October 2001

Potential Outcomes and Success Probability

Multiple SCSI, based Servers environments (locations) across a wide area

Existing Single SAN Fibre Channel environments with a number of FC Switches

Existing Single FC environment, that uses FC to connect mostly point to point and not invested in many, if any, switches

Existing Multiple FC environments (locations) on a single campus

Existing Multiple FC environments across a Wide Area

SCSI, and ATA based Host environments that need to grow into a network attachment

Single SCSI, based Server environments with needs to grow

Multiple SCSI, based Servers environments (locations) on a single campus

Multi Hosting environments with Storage Service Providers (SSPs)

Deployment Scenarios from Previous Slide Case FC Only FC&FCIP iFCP iSCSI iFCP&iSCSI iFCP Msg

1 High NA Low Low Med Helpful

2 High NA Low Low Med Helpful

3a High High High+ Low Very High Helpful

3b NA High High+ Low Very High Helpful

4 Low NA NA High Med NA

5 Low+ NA Low+ High Med Min Help

6a Med Med Med+ High Very High Min Help

6b NA Med Med+ High Very High Min Help

7 Hi->Med Med High Med>Hi Very High Helpful

Multiple SCSI, based Servers environments (locations) across a wide area

Existing Single SAN Fibre Channel environments with a number of FC Switches

Existing Single FC environment, that uses FC to connect mostly point to point and not invested in many, if any, switches

Existing Multiple FC environments (locations) on a single campus

Existing Multiple FC environments across a Wide Area

SCSI, and ATA based Host environments that need to grow into a network attachment

Single SCSI, based Server environments with needs to grow

Multiple SCSI, based Servers environments (locations) on a single campus

Multi Hosting environments with Storage Service Providers (SSPs)

Deployment Scenarios from Previous Slide Case FC Only FC&FCIP iFCP iSCSI iFCP&iSCSI iFCP Msg

1 High NA Low Low Med Helpful

2 High NA Low Low Med Helpful

3a High High High+ Low Very High Helpful

3b NA High High+ Low Very High Helpful

4 Low NA NA High Med NA

5 Low+ NA Low+ High Med Min Help

6a Med Med Med+ High Very High Min Help

6b NA Med Med+ High Very High Min Help

7 Hi->Med Med High Med>Hi Very High Helpful

Page 66: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 66

R

GbE

Port

010101I/O Block Data

LAN Data

010101

Intel and other vendors will haveIntel and other vendors will have

ONE Ethernet WireONE Ethernet Wire

forfor

ALL Storage & LAN TrafficALL Storage & LAN Traffic

I/O Adapters “Data Movers”

Page 67: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 67

Storage Functions/Applications

Current Functions/Applications Storage Consolidation Tape Backup Clustering Replication Disaster Recovery

New Capabilities with IP Storage SAN Extension QoS Security

Page 68: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 68

SAN SwitchRAID

Servers

Users

Tape SubsystemSAN Bridge

LAN-free Tape Backup

SAN Advantages for LAN-free Tape Backup: Removes backup traffic from the LAN Tape becomes SAN shared resource High performance SAN infrastructure SCSI attached via SAN bridge

Page 69: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 69

Fibre ChannelSCSI

GE, 10GE ( iSCSI, iFCP )

HBAs

TapeLibrary

RAID

NTServer

NTServer

RAID(Email)

TapeLibrary

LAN

Mission-Critical RAID(Oracle, ERP DB)

Backup Server :• Veritas Shared Storage Option • Tivoli Storage Manager

iSCSIServers

Remote Backup Application

Allows customers to move archiving off-site for higher disaster protection

Page 70: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 70

SAN SwitchRAID

Servers

Users

RAID

Heartbeat

Server Clustering

SAN Advantages for server clustering: Server access to common storage resources Failure of a single server still provides data access Scalable to > 30 servers in a cluster Simplified storage resource management

Page 71: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 71

NTServer

HBAs

RAID(Email)

TapeLibrary

TapeLibrary

RAID

NTServer

Fibre ChannelSCSI

GE, 10GE ( iSCSI, iFCP )

LAN

RAID

iSCSIServers

IP WAN

IP WAN Link (OC-3, T1, etc)

SAN Extension: Replication over WAN

Unified Management of Data Center and WAN storage routers Not vulnerable to disruption at a local SAN Leverage current infrastructure Expandable to iSCSI devices

Page 72: IP Storage Tutorial Presented 17 October 2001 by Marc Staimer, President & CDS – Dragon Slayer Consulting Ahmad Zamer, Sr. Product Line Marketing – Intel

17 October 2001 72

TCP/IP Layers

7

6

5

4

3

2

1

LAN/WANEthernet, token ring, ATM, Frame Relay, FDDI

IP

TCP/IPConnection oriented

UDPConnectionless oriented

FTP Telnet HTTP SNMP TFTP Process layer

Host to host layer

Internet layer

Network access layer

OSI Model TCP/IP Protocols TCP/IP layers