g2m research multi-vendor webinar #3: nvme over fabrics...
TRANSCRIPT
RESEARCH
G2M Research Multi-Vendor Webinar #3: NVMe Over Fabrics (NVMe-oF) Best PracticesNovember 6, 2018
2 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Webinar Agenda
9:00-9:05 Ground Rules and Webinar Topic Introduction(G2M Research)
9:06-9:25 Sponsoring Vendor presentations on topic (5 minute each) 9:26-9:39 Key Question 1 (2-minute question; 3 minutes response per
vendor) 9:40-9:41 Audience Survey 1 (2 minutes) 9:42-9:55 Key Question 2 (2-minute question; 3 minutes response per
vendor) 9:56-9:57 Audience Survey 2 (2 minutes) 9:58-10:11 Key Question 3 (2-minute question; 3 minutes response per
vendor)10:12-10:13 Audience Survey 3 (2 minutes) 10:14-10:18 Audience Q&A (8 minutes) 10:19-10:20 Wrap-Up
4 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Tom SpencerSr. Director, ProductMarketingSolarflarewww.solarflare.com
Panelists
Host/Emcee: Mike HeumannManaging PartnerG2M Researchwww.g2minc.com
John KimDirector, StorageMarketingMellanoxwww.mellanox.com
Laurent HendrichsSenior ProductLine ManagerBroadcomwww.broadcom.com
Praveen MidhaDirector, MarketingServer ConnectivityMarvellwww.marvell.com
5 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
What (and Why) is NVMe over Fabric
NVMe over Fabric (NVMe-oF) extends the NVMe block storage transport, which is normally native to PCI Express (PCIe), over various data center fabrics
NVMe-oF is important because it eliminates the protocol overhead inherent in SCSI-based storage protocols such as iSCSI (Ethernet) and Fibre Channel Protocol (FC)
NVMe-oF is a standard that is managed by NVMe.org (the NVM Express trade group)
6 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
NVMe over Fabric (NVMe-oF) Definitions and Flavors
NVMe transports running over Ethernet:– NVMe over RDMA over Converged Ethernet (NVMe-oRoCE)
– NVMe over iWARP (NVMe-oiWARP)
– NVMe over TCP (NVMe-oTCP)
NVMe transports running over Fibre Channel:– NVMe over Fibre Channel (NVMe-oFC)
NVMe transports running over InfiniBand:– NVMe over InfiniBand (NVMe-oIB)
9© 2018 Mellanox Technologies | Confidential
NVMe-oF Support Across Mellanox Product LineMore NVMe-oF Features Each Generation
ConnectX-410-100Gb (IB/Eth)
ConnectX-5E10-100Gb (IB/Eth)
PCIe Gen4
ConnectX-610-200Gb (IB/Eth)
BluefieldConnectX-5 + ARM
PCIe 4
Adapters
SoC
ConnectX-310/40/56Gb (IB/ Eth)
PCIe Gen3
2012 2018
Future Mellanox Products
Early Pre-Standard NVMe-oF
Demo NAB April
2015
5M IOPS 9us Latency
SC16 Demo
NVMe-oF Protocol Offload 2017
Single ASIC
NVMe-oF JBOF
Solution
10© 2018 Mellanox Technologies | Confidential
NVMe-oF Innovation with Protocol OffloadD
DR
4D
DR
4
PCIe Switch
NVME SSD
NVME SSD
Initiator x86
ConnectX-5
Initiator x86
ConnectX-5
DD
R4
DD
R4
PCIe Switch
NVME SSD
NVME SSD
Initiator x86
ConnectX-5
Initiator x86
ConnectX-5
SOC
6M IOPs, 512B block size 2M IOPs, 4K block side 50% CPU utilization ~15usec latency (without SSD)
8M IOPs, 512B block size 5M IOPs, 4K block side 0.01% CPU utilization ~5usec latency (without SSD)
ConnectX & Intel CPU no offload NVMe-oF offloadTarget Target
2 100Gb Initiators 2 100Gb Initiators
11© 2018 Mellanox Technologies | Confidential
Summary
Mellanox is a leader in NVMe-oF technology with ten plus customers already in production and tens more introducing their product in the near termOur products have supported NVMe-oF from its early beginnings
over RoCE and IB protocols and will soon add TCPWe are continuing to innovate new features for NVMe-oF like our
protocol offload and single ASIC JBOF solutions
14 Broadcom Proprietary and Confidential. © 2016 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|
The promise of NVMeoF
Perfo
rman
ce
Scalability
FC
NVMe
SAS/ SATA
iSCSI
NVMeoF
Direct Attach Ethernet
15 Broadcom Proprietary and Confidential. © 2016 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|
NVMeoF adoption status
Mature Silicon Hardware ecosystem
58800
Software ecosystem Early adopters
2016-18: Standardization and ecosystem development
2019: first sizeable NVMeoF deployments
2020: NVMeoF technology of choice for Ethernet networks
NVME AFAAI/Deep Learning
Composable Storage
SPDK
16 Broadcom Proprietary and Confidential. © 2016 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|
NVMeoF Architectures
Enterprise Cloud
BacksideFabric
Front sideFabric
NVMeJBOF/FBOF
StorageControllers
APPs
APPs
APPs
NVMe Scale Out Expansion
NVMe-oF NVMe-oF
Storage sideFabric
APPs
APPs
APPs
Rack Scale Disaggregation- HyperScale
NVMe-oF
Blocks ofStorage
Blocks ofCompute
Front sideFabric
EnterpriseArrays
APPs
APPs
APPs
Traditional SAN
NVMe-oF
17 Broadcom Proprietary and Confidential. © 2016 Broadcom. All Rights Reserved. "Broadcom" refers to Broadcom Limited and/or its subsidiaries.|
Storage Initiator• Offload services from x86
– Security, Erasure/RAID, Compression etc.• Manage storage separate from host
End to End NVMe-oF Solutions based on NetXtreme E and S Series
Storage Target• Provides efficient integrated solution for
NVMeoF• Models for enterprise scale up and
cloud scale out
FABRICRoCE and TCP/IP
S SeriesSmartNIC
E SeriesPerformance RNIC
or
JBOF Architecture
Storage
PCIe
Sw
itch
NVMe
NVMe
NVMe
NVMe
Stingray™1x100G
Stingray™1x100G
Scale Out Architecture
Con
nect
or
Stingray™
RESEARCH
SolarflareTom SpencerSr. Director, Product Marketingwww.solarflare.com
19 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
Five Reasons Why NVMe Over TCP is Cool
Tom Spencer | Senior Director, Product MarketingSolarflare
20 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
NVMe over Fabrics
NVMe-oTCPPlugin
Home
21 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
OptaneLatency~10us
0
5
10
15
20
25
30
35
40
45
512 4096
Late
ncy
us
Block Size
LinuxKernelNVMe
SPDK
SPDK+
Onload
LinuxKernel
SPDK
SPDK+
Onload
44%Lower
35%Lower
54%Lower
28%Lower
1. Performance
22 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
98% of installed servers connected with TCP
2% of installed servers connected with RDMA
2. Ease of Deployment
25 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
5. TCP Has Addressed Congestion
BeforeAfter
26 © 2018 Solarflare Communications, Inc. Confidential & Proprietary
5 Reasons Why NVMe-oF TCP is Cool
1. Performance
2. Ease of deployment
3. Runs on legacy networks
4. Ubiquity
5. TCP has addressed congestion
RESEARCH
MarvellPraveen MidhaDirector of MarketingServer Connectivity BUwww.marvell.com
M a r v e l l C o n f i d e n t i a l28
NVMe Server Software
Server Transport Abstraction
FibreChannel Infiniband FCoERoCEv2 iWARP
Storage Transport Abstraction
NVMe SSDs
TCP
Scaling our NVMe Requires a (Real) Network
Many options, plenty of confusion
Fibre Channel is the transport for the vast majority of today’s all flash arrays
FC-NVMe Standardized in Mid-2017
RoCEv2, iWARP and InfiniBand are RDMA-based but not compatible with each other
NVMe-oF RDMA Standardized in 2016
FCoE fabric is an option
NVMe/TCP - making its way through the standards process
M a r v e l l C o n f i d e n t i a l29
Enabling NVMeoTCP
NVMeoF alternative to iSCSI
NVMe-oF alternative to iSCSI for Storage Systems with PCIe NVMe SSDs
More efficient End-to-End operations as SCSI to NVMetranslations are eliminated
Co-Existence with NVMeoF
Co-exists with other NVMeoFtransport mechanisms
Transport selection may be based on h/w support and/or policy
No Forklift Offload
Enables NVMe-oF I/O operations in existing IP Datacenter environments
Software-only NVMe Host driver with NVMe-TCP transport
Offloaded NVMe-TCP for Target applications
M a r v e l l C o n f i d e n t i a l30
1 2
3
4
5
Universal RDMA + ToE
Ultimate in choice and investment protection with concurrent support for RoCE, RoCEv2 and iWARPand full offload for NVMe/TCP
FC-NVMe
Industry’s most accepted storage transport, now enhanced for NVMe.
Broad Spectrum of Ethernet and Fibre Channel
Connectivity16G and 32GFC , 10/25/40/50GbE
to host the most demanding Storage workloads and deliver
scalability
Seamless Storage Migration
Concurrent offload for NVMe-oF, iSER, iSCSI and FCoE for
Ethernet and FCP and FC-NVMe for Fibre Channel
enables seamless upgrade paths to next gen storage
End to End SolutionsInitiator and Target mode solutions, including SPDK that leverage years of storage and storage networking experience
NVMe-oF
Marvell NVMe-oF Solution
M a r v e l l C o n f i d e n t i a l31
NVMe-oF: Making the “Well-Informed” Choice?
Not “just” about “fabrics” performance Use CasesCulture and Install Base
32 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #1
Even though all of the major storage companies now support NVMe media in their AFAs, NVMe over Fabric is still in its infancy in AFAs. What do you think it will take for NVMe-oF to take off in AFAs?– Mellanox– Broadcom– Solarflare– Marvell
33 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #1
Has your organization deployed or are you planning to deploy NVMe-oF based storage in the near future? (check one):
• We currently have NVMe-oF based storage deployedin our organization: %
• We are planning on deploying NVMe-oF based storage within the next 12 months: %
• We are planning on deploying NVMe-oF based storage between 12 months and 24 months from now: %
• We are currently evaluating NVMe-oF based storage ,but we have not made a decision to deploy them yet: %
• We currently do not have a use case to deployNVMe-oF based storage in our organization: %
• I am unsure of my organization’s plans to deployNVMe-oF based storage: %
34 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #2
What do you think the primary applications with be that utilize NVMe-oF? When do you expect to see widespread deployment of these applications on NVMe-oF?– Broadcom– Solarflare– Marvell– Mellanox
35 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #2
What are your primary concerns when considering deployment of NVMe-oF (check all that apply)?
• We aren’t sure that we need NVMe-oF’s performance advantagesin the near future %
• Cost of NVMe-oF vs current storage networking solutions %
• Unsure of what use cases NVMe-oF makes sense for %
• NVMe-oF technology immaturity (standards, interoperability, etc.) %
• NVMe-oF ecosystem immaturity (arrays, switches, managementapplications, etc.) %
• Other %
36 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #3
How important will “smart NICs” be for NVMe-oF, and where are they best used?– Solarflare– Marvell– Mellanox– Broadcom
37 © 2018 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #3
What use cases do you see NVMe-oF utilized for in your organization over the next twelve months? (check all that apply):
• Databases/database applications: %
• Unstructured data/object storage: %
• Real-time analytics: %
• High-performance computing and modeling: %
• DevOps: %
• Other %
• None: %