How Next Generation NV Technology Affects Storage Stacks and Architectures

Marty Czekalski, Interface and Emerging Architecture Program Manager, Seagate Technology

Flash Memory Summit 2013 Santa Clara, CA

1

Outline

Overview of existing SW stacks and interfaces

New NVM devices NVM Programming Models

• Block, File, Persistent Memory (PM) Technical and Ecosystem Challenges

Conventional Storage Architecture

CPU

DRAM Memory

RAID Controller

Flash SSDs

PCIe Bus

Applications

OS Stack

IHV Driver

New Access

Methods

Typically File or Virtual IO, Load/Store

Logical Block IO

SAS or SATA interface Flash Translation

Layer and Management

Command, Queuing and DMA engines

Page Read, Page Write, Block Erase

Logical Block IO SCSI commands

SSD FW

Very Limited options

RAID or JBOD function

Flash parts

RAID or HBA FW

IHV Specific

Physical Block IO Conventional Drive FF and Bays

On Motherboard or add in PCIe CEM Card

PCIe RAID Aggregation Architecture

CPU

DRAM Memory

RAID Controller

Flash SSDs

PCIe Bus

Applications

OS Stack

IHV Driver

New Access

Methods

Typically File or Virtual IO, Load/Store

Logical Block IO

SAS or SATA interface

Flash Translation Layer and

Management

Command, Queuing and DMA engines

Page Read, Page Write, Block Erase

Logical Block IO SCSI commands

SSD FW

Very Limited options

RAID 0 or 5 function

Flash parts

RAID FW

IHV Specific

Physical Block IO Assembled on a PCIe CEM Card

CPU

DRAM Memory

Controller

Flash parts

PCIe Bus

Applications

OS Stack

Driver

On-load SSD Architectures

New Access

Methods

Typically File or Virtual IO, Load/Store

Logical Block IO

Physical Block IO

Flash Translation Layer and

Management

Command, Queuing and DMA

engines

Page Read, Page Write, Block Erase

Little or no firmware in controller, primarily HW state machines

More options

IHV Specific

PCIe CEM Card or drive FF (Express Bay or SATA Express)

NVMe and SCSIe Architectures

CPU

DRAM Memory

Controller

Flash parts

PCIe Bus

Applications

OS Stack

Driver

New Access

Methods

Typically File or Virtual IO, Load/Store

Logical Block IO

Physical Block IO

Flash Translation Layer and

Management

Command, Queuing and DMA

engines

Page Read, Page Write, Block Erase

Logical Block IO NVMe or SCSI commands

SSD FW

Limited options

NVMe or PQI

PCIe CEM Card or drive FF (Express Bay or SATA Express)

Linux IO Stack

Applications

VFS (e.g. block, Network,etc) Page cache

mmap/malloc

Block IO Layer

I/O Scheduler

SCSI Upper Layer

SCSI Mid Layer

SCSI Low Layer (IHV provided)

PCIe Device PCIe SSD, RAID Controller, HBA

IHV Block Driver

SCSI Express, NVMe, Proprietary

(e.g.)Optimized Page swap

Windows Storage Stack Applications

IO Subsystem

File System (e.g.NTFS)

Volume Snapshot

Volume Manager

Partition Manager

Storage Spaces

SCSI Class Driver

MPIO Driver

Storport

IHV Storport Driver

PCIe Device PCIe SSD, RAID Controller, HBA

SCSI Express, NVMe, Proprietary

SW RAID

Used for Virtually all Storage Interfaces

Multi-path

Storage Driver API

Promises of new NVM Technologies vs Flash

High performance and low cost • e.g. Phase Change, Memristor, RRAM • Near DRAM performance

Dramatically improved endurance Better scalability than current NVM solutions

• Replacement for flash (when cost/bit is close to parity)

NVM Programming Models

SNIA TWG (NVMP) preliminary specification, V1.0.0 Revision 5, available on SNIA.org

NVM Block Modes • Consistent with current block based storage stack

architectures, interfaces and protocols – Can use flash or new NVM technologies

NVM PM Modes • Utilizes memory which can be addressed using a

load/store model – New NVM technologies

NVM Block Interfaces

Source: SNIA NVM Programming Model V1.0.0 Revision 5

NVM Extensions

Leverage existing OS system constructs NVM Block Mode and File Mode Extensions

• Discovery and use of atomic write/read and discard features

• The discovery of granularities (length or alignment characteristics)

• Discovery and use of per-block metadata used for verifying integrity

• Discovery and use of ability for applications or kernel components to mark blocks as unreadable

• Potential use of memory mapped files

Relative Cost vs Time Chart C

ost

Time

NVM Persistent Memory (PM)

Memory capable of Load/Store operations Does not cause context switching NV DIMMS today – New NVM devices in the

future How to utilize PM in systems

NVM Persistent Memory (PM)Interfaces

Source: SNIA NVM Programming Model V1.0.0 Revision 5

Shifting the Persistence Boundary

CPU

Caches

DRAM

Storage

IO Subsystem Persistence Boundary

• Persistent • Durable • Snap Shots • Multi Access • Access Control • Error handling • Management • Security

Shifting the Persistence Boundary

CPU

Caches

DRAM

Storage

IO Subsystem

CPU

Caches

PM

Storage

IO Subsystem Persistence Boundary

• Persistent • Durable • Snap Shots • Multi Access • Access Control • Error handling • Management • Security

NVM PM Extensions

Again, build upon existing OS interfaces NVM PM Volume and PM File extensions

• Discovery and getting attributes; address ranges, connection channel, etc

• Memory map (w/options), sync, and discard functions

• Error handling

What’s the Right Physical Interface?

PCIe • Memory mapped IO latency is long compared to the

device access time DDR4

• Excellent speed but limited configurations and channels

• Existing virtual memory management assumptions may not map well into PM requirement

New Bus • Needs to be more expandable • Possibility of adding a management layer to better deal

with errors and media issue

Device Management

Endurance – Promised to be much better than Flash, but is it good enough? • Perhaps for some applications, but not all • Will need a high performance virtualization layer

– Integrate into processor virtual memory architecture – Even though devices capable of fine granularity writes,

management likely to be memory page granularity or larger – Should handle grown defects without rebooting

– Less frequent and simpler wear leveling approaches needed – SW flexibility vs hardware performance

ECC – Needs to be very low latency, similar to hamming codes • May need periodic scrubbing

Application adoption

Use of these new methods is not transparent to an application • E.g. Memory mapped files are not commonly used today • Applications need to worry about consistency

Cost and multi platform portability will continue to be the key factors in architectural choices • Initial adoption by lead users where return is worth the extra

effort/competitive edge

What features of storage are applications willing to compromise

Security concerns

Summary

Conventional architectures can benefit greatly in performance with new NVM technologies, but will only see adoption when cost is close to Flash

New NVM technologies will likely see early adoption as PM, requiring changes to applications and OSs to fully take advantage of performance capabilities

8/14/2013 22