paul luse, principal software engineer...for example: let’s say i want to write 4k to my nvm…....
TRANSCRIPT
Paul Luse, Principal Software Engineer
Agenda
• SNIA NVM Programming Model
• PMDK Overview
• Kit Comparison
• PMDK & SPDK Integration
• Potential Use Cases
2
3
NVDIMMs
UserSpace
KernelSpace
Standard
File API
NVDIMM Driver
Application
File System
ApplicationApplication
Standard
Raw Device
Access
Load/Store
Management Library
Management UI
Standard
File API
pmem-AwareFile System
MMU
Mappings
4
• PMDK is a collection of libraries
• Developers pull only what they need
• Low level programming support
• Transaction APIs
• Fully validated
• Performance tuned.
• Open Source & Product neutral
http://pmem.io/
5
Support for
volatilememory
usage
libmemkind
Low level
support for
local persistent
memory
libpmem
Low level support for remote access to persistent memory
librpmem
In Development
Interface to create
arrays of pmem-
resident blocks, of
same size,
atomically updated
Interface for persistent
memory allocation,
transactions and
general facilities
Interface to create
a persistent
memory resident
log file
libpmemblklibpmemlog libpmemobj
C++ C PCJ* Python
* Persistent Collections for Java
6
PMDK SPDK
IO model Synchronous Asynchronous
Requires app changes Yes Sometimes
Data Access Granularity Byte Block
Data Access Style Memory I/O
Media Persistent Memory SSD, PM, Kernel Devices (AIO)
Open Source Yes Yes
7
For example: Let’s say I want to write 4K to my NVM…. (skipping lots of details)
PMDK SPDK
1. Integrate libpmem with application
2. Memory map my PM file on my PM-aware file system using pmem_map_file()
3. Perform assignments directly on the memory addresses
4. Make persistent with pmem_persist()
The memory location represents direct access to the media.
1. Integrate SPDK with application2. Allocate DMA safe memory buffer
and update accordingly.3. Use block device layer API
spdk_bdev_write()
4. Poll for I/O completion
Once the IO has been accepted by the NVMe controller, the memory location is DMA’d by the NVMe controller to the media and a completion is posted to the application.
Intel® Builders 8
PMDK & SPDK independent Integration
• Integrate PMDK w/Application independent of SPDK
• Separate path to faster media via SNIA PM
• Possible Use Cases:
• SPDK path in application already in production, but application has other use cases for high speed persistence,
• Cache
Application
Drivers
<SPDK Defined Block API>
<Driver Specific API>
I/O
PM
by
tes
fs
PMDK
con
tro
l
Block Device Layer
SPDK
Intel® Builders 9
PMDK & SPDK Integration via libpmemblk• Bdev module leveraging libpmemblk to access
persistent memory as blocks
• Libpmemblk addresses block atomicity requirement in software to protect from torn writes
• Bdev API out the top, persistent memory SNIA programming model out the bottom, directly to media.
• Possible Use Cases:
• Stepping stone to PM access via SNIA PM
• Separate path to faster media via block access
• Block cache, block based metadata
Block Device Layer
Application
Drivers
<SPDK Defined Block API>
Libpmemblkbdev module
<Driver Specific API>
I/O
PM
by
tes
Other bdev Modules
Intel® Builders 10
PMDK & SPDK complete Integration• Libpmem integrated into the bdev layer for use by
other bdev functions and bdev modules
• Enables use of PM as persistent storage replacing DRAM in many cases.
• Possible Use Cases:
• Optimal use of SPDK fast path and PM “faster” path
• Data cache
• Metadata storage of any kind
• Write ahead log
Block Device Layer
Application
Drivers
<SPDK Defined Block API>
libpmem
<Driver Specific API>
I/O
PM
by
tes
bbdevmodules
Bdevfunctions
11
Backup
12
• libpmem• Detects types of flush instructions supported by the CPU, and uses best instructions for the platform to
create performance-tuned routines for copying ranges of persistence memory. • libpmemobj is the core library that supports persistent memory allocations, transactions and relies on lower
level libraries for data persistence support.• librpmem:
• Provides low-level support for remote access to persistent memory (pmem) utilizing RDMA-capable NICs. This library can be used to replicate remotely a memory region over RDMA protocol. It utilizes appropriate persistency mechanism based on the remote node’s platform capabilities.
• libpmemlog• support specific usages and are tuned for these usages. For example, libpmemlog is useful where an
application frequently appends to log file. • libpmemblk
• useful to manage application block cache residing in persistent memory.• libpmemlog, libpmemblk, libpmemobj libraries are built on top of libpmem and support transactions.