abhinav singh software technical consulting engineer › download › ... · abhinav singh software...
TRANSCRIPT
Abhinav Singh
Software Technical Consulting Engineer
SPDK, PMDK & VTune™ Amplifier Summit
Agenda
• Introduction to SPDK Architecture
• Data Telemetry
• Data Collection and Metrics
• Conclusion
SPDK, PMDK & VTune™ Amplifier Summit
Drivers
StorageServices
StorageProtocols
iSCSI Target
NVMe-oFTarget
vhost-scsiTarget
NVMe Devices
Blobstore
NVMe-oFInitiator
Intel® QuickDataTechnology
Driver
Block Device Abstraction (BDEV)
Ceph* RBD
Linux* Async IO
Logical Volumes
3rd Party
NVMe
NVMe PCIeDriver
vhost-blkTarget
BlobFS
Integration
RocksDB*
Ceph*
Core
ApplicationFramework
GPT
PMEMvirtio-
scsi
QEMU
SPDK Architecture
Intel®VTune™
SPDK, PMDK & VTune™ Amplifier Summit
Supported SPDK usage models
Application services:
SPDK vhost-scsi for serving optimized block storage to VMs
SPDK NVMe for optimized access to locally attached storage
Disaggregated storage:
NVM Express* over Fabrics
iSCSI target
SPDK, PMDK & VTune™ Amplifier Summit
SPDK data processing using Intel® Data Direct I/O Technology (Intel® DDIO)
CPU Socket
CPU Cores ME
MO
RY
CO
NT
RO
LL
ER
RA
M
1
LLCI/O Data
1
5
2
RXd
Disk
4 3
6
1. Core writes I/O command descriptor
2. Disk reads descriptor to get buffer address
3. Disk writes I/O data
4. Disk writes I/O command descriptor
5. Core reads I/O command descriptor (polling)
6. Core reads IOPs and performs some action
SPDK, PMDK & VTune™ Amplifier Summit
Telemetry points
CPU SocketCPU Socket
CPU Cores ME
MO
RY
CO
NT
RO
LL
ER
1
LLCPacketRXd
Disk
15
23
6 RA
M
4
Core communicates with storage through PCIe MMIO transactions Intel DDIO makes LLC the
primary target ofDMA operations
Core I/O data processing
Memorybandwidth
Writing back descriptors may result in partial PCIe transaction
Intel® VTune™ integrationwith SPDK
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Summary
Get an overall I/O performance data Notice I/O
performance imbalance among
SSDs
We can also seehow each device
performs individually
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Summary
See throughput distributionper each device
Set reasonable performance targets and get throughput utilization per each range
All metrics can be seen on timeline within thread quanta…see next slides
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Bottom-up
in-thread activitycolored according
to Throughput utilization levels SPDK Effective time: CPU usage is accounted separately for spin-wait loops
I/O Statistic: IOPs and Throughput
PCIe bandwidth with traffic breakdown per physical device
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Bottom-up
Locate recessions and filter in to see I/O performance changes
Check which code was executed and caused I/O communication
being dropped for ~160ms!
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Platform view
PCIe bandwidth with traffic breakdown per physical device
Intel DDIO missesresulted in write back
to RAM
MMIO traffic. AvoidReads and control
Writes
DRAM bandwidth
Socket interconnect traffic
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Platform view
huge writeback traffic to host memory due to misses in IO-ways
SPDK, PMDK & VTune™ Amplifier Summit
Analyzing results – Platform view
Ʃ= 93,362,354
Almost 1:1 between doorbell signaling
and total number of IO operations. IO* 46,681,177
SPDK, PMDK & VTune™ Amplifier Summit
Easy
Advance
Calibrate I/O bandwidth with expected performance
Check for unexpected cross-NUMA traffic
Check for unexpected PCIe MMIO traffic
Optimize workload to make use of Intel DDIO efficiently
Analyzing performance with Intel® VTune™
SPDK, PMDK & VTune™ Amplifier Summit
Conclusion and Call to ActionSPDK proves to be useful to accelerate storage applications equipped with NVMe based devices
Run Disk I/O analysis in Intel® VTune™ Amplifier 2019 to ensure that I/O solution can deliver the performance required for the intended usage model
Call to action:
Welcome to use SPDK and contribute into SPDK community
Check out SPDK at http://spdk.io Download
Intel® VTune Amplifier Performance Profilersoftware.intel.com/vtune
Documentation:
User guide: SPDK IO Data Viewsoftware.intel.com/vtune-amplifier-help-spdk-code-analysis
IO Issues: Remote socket accesssoftware.intel.com//vtune-amplifier-cookbook-i/o-issues-remote-socket-accesses
SPDK, PMDK & VTune™ Amplifier Summit
Legal Disclaimer & Optimization Notice
INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks.
Copyright © 2018, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
Optimization Notice
Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804
SPDK, PMDK & VTune™ Amplifier Summit
BACKUP
SPDK, PMDK & VTune™ Amplifier Summit
cd <path_to_SPDK>
scripts/gen_nvme.sh
[Nvme]
TransportID "trtype:PCIe traddr:0000:00:0e.0" Nvme0
paste the ouput to test/lib/bdev/bdev.conf.in
cd test/lib/bdev/bdevperf
run SPDK sample application
sudo ./bdevperf -c ../bdev.conf.in -q 32 -s 4096 -w rw -M 50 -t 10
cd <path_to_Vtune/bin64>
run VTune
sudo ./amplxe-cl -collect hotspots -knob enable-user-tasks=true -- <app>