investigating serial attached scsi (sas) over tcp ( tsas )
DESCRIPTION
Investigating Serial Attached SCSI (SAS) over TCP ( tSAS ). UCCS Master’s Project Deepti Reddy. Outline of the Talk. Introduction to tSAS and related technology Motivation and Challenges tSAS Design Performance Evaluation of a Mock Application Lessons Learnt and Future Directions - PowerPoint PPT PresentationTRANSCRIPT
Investigating Serial Attached SCSI (SAS) over TCP (tSAS)
UCCS
Master’s Project
Deepti Reddy
tSAS/sreddy 2
Outline of the Talk Introduction to tSAS and related
technology Motivation and Challenges tSAS Design Performance Evaluation of a Mock
Application Lessons Learnt and Future Directions Conclusion
11/3/2011
tSAS/sreddy 3
SCSI vs SASSCSI (Small Computer System Interface)
• The SCSI protocol is an application layer storage protocol. • It's a standard for connecting peripherals to your computer
via a standard hardware interface, which uses standard SCSI commands.
SAS (Serial Attached SCSI)
• SAS is the successor of SCSI technology• Initially introduced at 3Gb/s in 2004• Currently, supports 6Gb/s• 12Gbps by 2012• Significantly increased the available bandwidth offered by
legacy SCSI storage systems.• Use of expanders increases the scalability over legacy SCSI
11/3/2011
4
SCSI vs SASSCSI SAS
Topology Parallel Bus Serial BusReduces Noise (Better signal integrity)
Speed 3.2 Gbps 3 Gbps, 6Gbps, 12Gbps
Distance 1 to 12 meters 8 metersNumber of Targets 14 devices 128 expanders.
> 16,000 with cascaded expanders
Devices SCSI only SAS & SATAUsed widely in Data Centers today
Connectivity Single Port Dual PortFail-over/fail-back
Drive Form Factor 3.5” 2.5”Cost Low Medium11/3/2011
tSAS/sreddy
5
iSCSI iSCSI protocol describes a
means of transporting of the SCSI packets over
TCP/IP The iSCSI specification is
revised and updated by the Internet Engineering task Force (IEFT).
Work by S. Aiken, D. Grunwald, A. Pleszkun and J. Willeke shows the performance of a commercial iSCSI software implementation compared quite favorably with fibrechannel [7]
iSCSI protocol layering model11/3/2011 tSAS/sreddy
tSAS/sreddy 6
Typical SAS Topology
X X X
X
- DISK DRIVES
- EXPANDERS
DriverSAS CONTROLLER(HOST BUS ADAPTER)
~8m
11/3/2011
tSAS/sreddy 7
Motivation & Challenges Overcome the distance and scalability
limitations of traditional Serial Attached SCSI (SAS) by using tSAS.
Like iSCSI, tSAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations
Provide research results for future industry specification for tSAS and iSCSI.
11/3/2011
tSAS/sreddy 8
Related Work
Michael Ko’s patent at IBM Research-Almaden on Serial Attached SCSI over Ethernet proposes a very similar solution to the tSAS solution provided in this project.
iSCSI specification (SCSI over TCP) itself is similar to a tSAS solution (SAS over TCP). The iSCSI solution can be heavily leveraged for a tSAS solution.
The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a tSAS solution.
11/3/2011
tSAS/sreddy 9
tSAS Topology
X
X
- DISK DRIVES
- EXPANDERS
SCSI DrivertSAS CONTROLLER/HBA
- TCP link- SAS link
SCSI DrivertSAS
HBA
11/3/2011
tSAS/sreddy 10
X
X
- DISK DRIVES
- EXPANDERS
SCSI DrivertSAS CONTROLLER/HBA
- TCP link- SAS link
tSASHBA
SCSI Driver
tSAS Topology
11/3/2011
tSAS/sreddy 11
Goals of the project Investigate how tSAS can be implemented Design and develop a mock tSAS application
− Write a client program using C/C++ that sends a set of tSAS commands to a server.
− Write a server program using C/C++ that receives a set of tSAS commands, processes them and responds to the client with tSAS responses.
Evaluate the tSAS solution and compare with a mock iSCSI client/server application.
11/3/2011
tSAS/sreddy 12
Software and Hardware solutions for tSAS implementations
Software based tSAS solution− Cheaper than a hardware based solution− All tSAS processing is done by the processor
and TCP/IP operations are also executed by the CPU
− NIC is merely an interface to the network TCP Offload Engine (TOE) solution
− NICs with integrated TOEs have hardware built into the card that allows the TCP/IP processing to be done at the interface
Hardware based tSAS solution− In a hardware-based tSAS environment, the
initiator and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and tSAS processing
11/3/2011
tSAS/sreddy 13
tSAS Approach The tSAS Request is initially sent by the
tSAS Initiator to the tSAS Target over TCP. The tSAS Target strips off the TCP header
and sends the SAS frames using the SAS Initiator block on the tSAS Target to the SAS expander/drive.
The SAS expander/drive sends SAS frames to the tSAS Target.
Finally, the tSAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the tSAS Initiator.
11/3/2011
14
tSAS Message Format
The above shows how a legacy SAS header and data are embedded in an Ethernet frame.
iSCSI uses the same approach where the iSCSI header and data are encapsulated in an Ethernet frame.
Ethernet Header
IP Header
TCP Header
tSAS Header
tSASData
Ethernet Trailer
TCP SegmentIP Datagram
Ethernet Frame
11/3/2011 tSAS/sreddy
15
tSAS Approach
The above diagram shows a typical tSAS (SSP Write ) Request & Response sequence diagram.
iSCSI uses a similar approach where the back-end between the iSCSI target and SCSI drives uses the legacy SCSI protocol.
tSAS Request
tSAS Initiator
tSAS Target
SAS Expander
Open Address Frame
Open Accept
SAS/SSP Request Frame
Xfer Rdy Frame
.
.Data Frame
Data Frame Data Frame
tSAS Xfer Rdy Frame
.
.Data Frame
11/3/2011 tSAS/sreddy
SSP Response
FrametSAS
Response Frame
tSAS/sreddy 16
Primitives− Primitives are special 8b/10b encoded
characters that are used as frame delimiters, for out of band signaling, control sequencing in SAS
− Most primitives can be conveniently ignored on the tSAS side except a few such as Broadcast primitives used to initiate discovery
− SAS primitive can be encapsulated in an Ethernet frame
Discovery− Discovery in tSAS will be similar to SAS and
will be accomplished by sending Serial management protocol (SMP) commands over TCP to the initiators and expanders downstream to learn the topology.
tSAS Approach
11/3/2011
tSAS/sreddy 1711/3/2011
tSAS Approach
SAS Primitives
tSAS/sreddy 18
SMP Request
tSAS Client
SMP Initiator
on Expander
SMP Target on Expander
Open Address FrameOpen
Accept
SMP Request
SMP Response
CloseSMP Response
tSAS Approach
SMP Request & Response Sequence Diagram
11/3/2011
tSAS/sreddy 19
Test benches used for performance calculations
tSAS and iSCSI Mock Application− The client application and server application
run on two different Windows machines connected using a NetGear ProSafe Gigabit Switch
Intel Core 2 Duo CPU 2.53 GHz, 3.45 GB RAM Intel 82567LM Gigabit Network Adapter
− Client makes Read/Write requests and the server processes and responds to requests made by the Client
11/3/2011
Windows Machine A(Running
Client App)
Windows Machine B(Running
Server App)
NetGearSwitch
tSAS/sreddy 20
Test benches used for performance calculations
Legacy SAS− A 6 Gbps SAS Host Bus Adapter in a PCIe slot
of a Super Micro server− A 6 Gbps SAS target (Seagate SAS drive)
connected to the Host Bus Adapter− IOMeter in Windows and VDBench in RHEL
used to generate Read/Write IOs and measure performance
IOMeter and VDBench are IO generator tools used to measure IO performance of a storage system
− A LeCroy SAS Protocol Analyzer placed between the HBA and the SAS Target
11/3/2011
tSAS/sreddy 2111/3/2011
X
X
- SAS DISK DRIVE- EXPANDER
DriverSAS CONTROLLER(HOST BUS ADAPTER)
SAS Protocol Analyzer
SAS Protocol Analyzer
Test benches used for performance calculations
2211/3/2011 tSAS/sreddy
Test benches used for performance calculations
Legacy iSCSI− Two windows machines/servers were used
connected using a NetGear ProSafe Gigabit Switch
− On one machine the StarWind iSCSi Initiator was running
− On another machine, the KernSafe iSCSi target software was used to create a iSCSI target
− IOMeter was used to send Read/Write requests from the iSCSI Initiator to the iSCSI Target and measure performance
− Wireshark used to capture Network Traffic
iSCSI Initiator on Windows
Machine A
iSCSI Target on Windows
Machine B
NetGearSwitch
tSAS/sreddy 23
tSAS vs iSCSI Mock Application at 10 Mbps
11/3/2011
0 200 400 600 800 1000 12000
5001000150020002500
iSCSI vs tSAS READ Completion Time at 10 Mbps
tSAS READ Completion timeiSCSI READ Completion Time
Transfer Size (Kilobytes)
Time (Milliseconds)
0 200 400 600 800 1000 1200 1400 1600 1800 2000 22000
2000
4000
6000
8000
tSAS vs iSCSI Write 10Mbps
tSAS Write 10MBpsiSCSI Write 10MBps
Transfer Size (Kilobytes)
Time (milliseconds)
tSAS/sreddy 24
tSAS vs iSCSI Mock Application at 100 Mbps
11/3/2011
0 200 400 600 800 1000 1200 1400 1600 1800 2000 22000
100200300400500600700
tSAS vs iSCSI Read 100 Mbps
tSAS Read 100 MbpsiSCSI Read 100 Mbps
Transfer Length (Kilobytes)
Time(Milliseconds)
0 200 400 600 800 1000 1200 1400 1600 1800 2000 22000
200400600800
1000
tSAS vs iSCSi Write 100 Mbps
tSAS Write 100 MbpsiSCSI Write 100 Mbps
Transfer Length (KB)
Time(Milliseconds)
tSAS/sreddy 25
tSAS vs iSCSI Mock Application at 1 Gbps
11/3/2011
0 200 400 600 800 1000 1200 1400 1600 1800 2000 22000
1020304050
tSAS vs iSCSI Read 1 Gbps
tSAS Read 1 GbpsiSCSi Read 1 Gbps
Transfer Length (KB)
Time(Milliseconds)
0 200 400 600 800 1000 1200 1400 1600 1800 2000 22000
1020304050607080
tSAS vs iSCSI Write 1 Gbps
tSAS Write 1 GbpsiSCSI Write 1 Gbps
Transfer Size (KB)
Time(Milliseconds)
tSAS/sreddy 26
tSAS 512K Read at 10 Mbps, 100 Mbps and 1 Gbps
11/3/2011
0 200 400 600 800 1000 12000
200
400
600
800
1000
1200
Time for READ Completion with transfer size of 512K at 10 Mbps, 100Mbps and 1Gbps
Time for READ Compeltion
Connection Rate (Mbps)
Time (Milliseconds)
27
Lessons Learnt/Difficulties encountered
Using several iSCSI Target Softwares that were not licensed/purchased made it challenging since several features were either not supported or did not work on the unlicensed version. Even using KernSafe iSCSI target, I was not
able to create a iSCSI RamDisk Target or a iSCSI Hard disk target. I used a USB Flash drive as the iSCSI target
− Subtracted the delay at the Flash Drive from the iSCSI IOMeter performance results
− Delay of Flash Drive calculated by running IOMeter to the SCSI device on the target machine
11/3/2011tSAS/sreddy
tSAS/sreddy 28
Jumbo frames The NetGear ProSafe switch is a Gigabit
switch that supports Jumbo Frames. However, I could not get jumbo frames to work since the Network Adapter on the Windows machines did not support Jumbo frames
One needs cabling and adapters as well that support Jumbo frames to use a switch that supports Jumbo frames
Understanding the SAS protocol and tools is very involved− Took substantial time to understand and learn
the workings of SAS as well as the tools
11/3/2011
Lessons Learnt/Difficulties encountered
tSAS/sreddy 29
Future Work More data points
− The tSAS mock application can be run using a faster switch with connection rates greater than 1 Gbps to get more data points (40G/100G Ethernet)
Piggybacking− Response frames are piggybacked with the last
DATA frame or a DATA frame is piggy backed with a request frame. This would increase performance
Jumbo frames − Can be used to increase the amount of DATA that is
passed from the initiator and target per Ethernet packet improving the performance results.
The Storage Associations can be motivated with similar work to fuel the conception of a tSAS specification
tSAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation
11/3/2011
30
Conclusion tSAS is a viable solution tSAS will be faster than a similar iSCSI
implementation− Overhead in tSAS smaller than in iSCSI
Request, Response and Transfer Ready frames in tSAS smaller than Request, Response and Ready to Transfer frames in iSCSI
− Back-end of tSAS will be legacy SAS− tSAS more visibly faster at larger transfer
sizes than iSCSI− tSAS more reliable at smaller transfer sizes at
10 Mbps, 100 Mbps and 1 Gbps due to higher TCP retransmission rate at larger sizes
− tSAS hardware solution would be the fastest vs a software solution
− tSAS would perform better at smaller transfer sizes
tSAS will overcome the distance limitation of legacy SAS
11/3/2011 tSAS/sreddy
tSAS/sreddy 31
References [1] T10/1760-D Information Technology – Serial Attached SCSI – 2 (SAS-2), T10, 18 April 2009, Available from http://www.t10.org/drafts.htm#SCSI3_SAS [2] Harry Mason, Serial attached SCSI Establishes its Position in the Enterprise, LSI Corporation, available from http://www.scsita.org/aboutscsi/sas/6GbpsSAS.pdf [3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner, RFC 3720 Internet Small Computer Systems Interface (iSCSI) Specification IETF, April 2004, available from http://www.ietf.org/rfc/rfc3720.txt [4] Cai, Y.; Fang, L.; Ratemo, R.; Liu, J.; Gross, K.; Kozma, M.; A test case for 3Gbps serial attached SCSI (SAS) Test Conference, 2005. Proceedings. ITC 2005. IEEE International, February 2006, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1584027
11/3/2011
tSAS/sreddy 32
References [6] Kalmath Meth, Julian Satran, Design of the iSCSI Protocol,Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1194848&tag=1 [7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke, A Performance Analysis of the iSCSI Protocol 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, available from http://www.storageconference.org/2003/papers/20-Aikens-Performance.pdf [8] M. Rajagopal, E. Rodriguez, R. Weber, RFC 3821 Fibre Channel over TCP/IP (FCIP) standard, IETF, July 2004, available from http://tools.ietf.org/html/rfc3821
[9] BLi Bigang, Shu Jiwu, Zheng Weimin, SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS *
Department of Computer Science and Technology, Tsinghua University, Beijing China, 2005
11/3/2011
tSAS/sreddy 33
References [10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. McDaniel, M. Vilayannur, Design, Implementation and Evaluation of Security in iSCSI-based Network Storage Systems, StorageSS '06 Proceedings of the second ACM workshop on Storage security and survivability, October 2006, available from http://portal.acm.org/citation.cfm?id=1179564
[11] 1Gbps and 10Gbps Ethernet Server Scalability, NetApp, available fromhttp://partners.netapp.com/go/techontap/matl/downloads/redhat- neterion_10g.pdf
[12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM INTERFACE (SAS) vOVER ETHERNET, United States Patent Application 20080228897, 09/18/2008 available from http://www.faqs.org/patents/app/20080228897
[13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T10, 2011/06/13, available from http://www.t10.org/members/w_spc4.htm
[14] Broadcom iSCSI HBA FAQ available from http://www.broadcom.com/collateral/faq/iSCSI-FAQ100-R.pdf
11/3/2011