cosmos openssd: a pcie-based open source ssd platform · pdf fileerror correction code bch 32...

Cosmos OpenSSD:

A PCIe-based Open Source SSD Platform

Yong Ho Song1, Sanghyuk Jung1,

Sang-Won Lee2, Jin-Soo Kim2

Hanyang University1, Sungkyunkwan University2, Korea

Flash Memory Summit 2014

Santa Clara, CA

1

OpenSSD Introduction


Santa Clara, CA

2

What’s the OpenSSD Project

Open-source SSD platform for research and

education on the SSD technology since 2011

New “OpenSSD platform” for developing SSD

firmware, controller hardware, and host

software

Contribution

• Indilinx (Merged to OCZ in 2012)

• HYU (Hanyang University), Korea

• SKKU (Sungkyunkwan University), Korea


Santa Clara, CA

3

Why OpenSSD?

Solve your problem in a real system

Share your solution with people in society

Design your own SSD controller, if possible

Contribute to “open” community

Use it as a PC disk

Play for fun


Santa Clara, CA

4

OpenSSD Project History

Jasmine OpenSSD (2011)

• SSD controller: Indilinx Barefoot (SoC w/SATA-2)

• Firmware: SKKU VLDB Lab

• Users from 10+ countries

• 10+ papers published


Santa Clara, CA

5

Barefoot Controller

SoC

SATA-2

Interface

SATA-2 Interface

NAND Flash Memory

(32GB/module)

OpenSSD Project History

Cosmos OpenSSD (2014)

• SSD controller: HYU Tiger3 (FPGA w/PCIe Gen2)

• Firmware: HYU ENC Lab, SKKU VLDB Lab

• Users from ?? countries (at least one)

• ?? papers to be published (at least three)


Santa Clara, CA

6

SSD Controller In FPGA

External PCIe

Interface

NAND Flash Memory

(256GBs/module)

Comparison among the Platforms


Santa Clara, CA

7

Jasmine

OpenSSD

Cosmos

Prototype (Tiger2)

Cosmos

OpenSSD (Tiger3)

SSD

Controller

Indilinx Barefoot

(SoC)

HYU Tiger2

(FPGA)

HYU Tiger3

(FPGA)

Year 2011 2012 2014

Host Interface SATA-2 PCIe Gen1.1

(AHCI Subset)

PCIe Gen2

(AHCI Subset)

Storage

Capacity 128 GB 512 GB 512 GB

NAND Data

Interface Asynchronous Asynchronous Synchronous

DRAM

Capacity 64 MB 512 MB 1 GB

OpenSSD Project Homepage


Santa Clara, CA

8

http://www.openssd-project.org

Cosmos OpenSSD Platform

Features


Santa Clara, CA

9

Design Objective

A simple NAND storage platform to be used

for

• Firmware development and evaluation

• Controller architecture exploration

• New controller design (e.g. Intelligent SSD)

An open-source SSD platform with AP-class

CPUs and other hardware resources


Santa Clara, CA

10

Cosmos SSD Hardware Features


Santa Clara, CA

11

FPGA Xilinx Zynq-7000 series

MCU Type CortexTM- A9

Clock Frequency 667 MHz

Storage Total Capacity 512 GB

NAND Organization 4-Channel / 8-Way

DRAM Device Interface DDR3 (533 MHz)

Total Capacity 1 GB

Bus System AXI-Lite (Bus width: 32 bits)

Storage Data AXI (Bus width: 64 bits, Burst Length: 256)

SRAM 256 KB (FPGA Internal)

Error Correction Code BCH 32 bits/2 KB

Host Interface PCI-Express Gen2 4-Lane (2 GB/s)

Power Measurement NAND Flash and Board Power Measurement

(External ADC Module or NI DAC)

NAND Flash


Santa Clara, CA

12

Vendor / Model SK Hynix / H27QDG8VEBIR-BCB

Device Interface Synchronous (ONFI 2.2)

Cell Technology MLC (2 Bits/cell)

Capacity 16 GBs/package

Page Size Data Area 8192 Bytes/page

Spare Area 640 Bytes/page

Dies per Package 4 dies (QDP)

Data Cycle 20 ns (DDR: 10 ns)

Page Read 70 us (typical)

Page Program 1400 us (typical)

Block Erase 3.5 ms (typical)

Speed NV-DDR100

NAND Flash Memory Specification

DRAM


Santa Clara, CA

13

Vendor Samsung (K4B4G1646B-HCK0)

Device Interface DDR3-1066F

DRAM Device Bus Width 16 Bits

Capacity 1 GB (512 MBs x2)

CAS Latency 7 Cycles

CAS Write Latency 6 Cycles

RAS to CAS Delay 7 Cycles

Precharge Time 7 Cycles

Throughput 8.5 GB/s

DRAM Memory Specification

Cosmos SSD Platform Board


Santa Clara, CA

14

1

2

3

4

4

5

6

7

8

9

No. Information

1 Xilinx Zynq-7000

(HYU Tiger3 Controller)

2 DDR3 (512 MB x2)

3 NAND SODIMM x2

4 External PCIe x2

5 USB JTAG

6 UART

7 USB 2.0 OTG

8 SD Card

9 Power Measurement

Connector

HYU Tiger3 Controller


Santa Clara, CA

15

ECC Engine

NAND Flash Way Controller



NAND Flash Channel

Controller

NAND Flash Buffer

Controller

NAND Flash Arrays

NAND Flash Arrays

NAND Flash Arrays

NAND Flash Arrays

NAND Storage Controller

ARM A9 Dual-Core

External Interface

QSPI Flash Controller

DDR3 Memory Controller

Central Interconnect

FPGA to Memory

Interconnect

ARM A9 Processor System

ZYNQ-7000 FPGA (XC7Z045-3FFG900)

256 Mb QSPI

RS232

AXI Bus

DDR3-1066 1 GB DRAM

DMA TX Engine

DMA RX Engine

Host I/F Controller

7 Series PCI-Express

Core PCI-Express

Gen2 4-Lane

Host

PCI-Express Engine with SG-DMA

SG DMA Controller

RX FIFO

TX FIFO

Performance Monitor

Page Buffer

HYU Tiger3 Controller

Storage Performance Monitor


Santa Clara, CA

16

ECC Engine NAND Flash

Way Controller #0

NAND Flash Way Controller #2


NAND Flash Channel Controller

NAND Flash Buffer Controller

NAND Flash Arrays

NAND Flash Arrays

NAND Flash Arrays

NAND Flash Arrays

Storage Controller


…

Performance Monitor

Page Buffer

Storage Requests

Program Latency

Read Latency

Channel Utilization

Buffer Occupation

GC Invocation

GC Latency

Software Execution Latency

Bit Error Status

∽

Statistics Registers

Statistics on the performance and utilization of hardware resources

Use of Performance Monitor


Santa Clara, CA

17

Performance

Monitor Channel

Controller

BCH Engine

System Bus

Memory

Controller

Host I/F

Controller

(PCIe)

UART I/O Ports CPU I2C / SPI

DRAM NAND

Array

Storage Controller

Application

(C/C++)

Software

Host System

Hardware

RTL

(Verilog/VHDL)

Software module

performance

Storage

performance

Hardware module

performance

Cosmos OpenSSD Platform

Example: Operation Time

Breakdown


Santa Clara, CA

18

Host Buffer

(DRAM) NAND

Data Register

Page

Page

Page ●

● ●

tPROG

BUF2NAND

HOST2BUF

CFLRU

Re

sp

on

se

Tim

e

tPR

OG

Page Number

LSB

MSB (FAST)

MSB (SLOW)

AVG LSB MSB (SLOW) MSB (FAST)

Available FTLs (under GPL)

TutorialFTL (by HYU ENC Lab)

• Page-mapping FTL w/o GC

• Device initialization and interface definition

GreedyFTL (by HYU ENC Lab)

• Page-mapping FTL w/ Greedy GC

DramDisk (by HYU ENC Lab)

• DRAM Disk for measuring maximum read/write

performance (PCIe + DRAM)


Santa Clara, CA

19

Common Operation Flow


Santa Clara, CA

20

Device Initialization

Firmware Start ()

Call NAND_read()

IDLE OP?

Background OP

CMD READY?

READ?

Call NAND_write()

No Yes

Yes No

Available Not

Available

Command Parsing

Select a page to op

BUF HIT?

Call buf_read()

BUF HIT?

Yes

Call buf_write()

Yes

CMD END?

No Yes

No No

Used in TutorialFTL and GreedyFTL

Host Software

AS-IS

• Makes the storage work as a drive

• Handles read/write requests from kernel

• Needs to install storage-specific device driver

– Windows7 64-bit device driver (by HYU ENC Lab)

– Linux 3.2.X kernel device driver (by HYU ENC Lab)

TO-BE

• Compatible with PCIe system drivers

– AHCI support

– NVMe support


Santa Clara, CA

21

Some Use Cases


Santa Clara, CA

22

When compared to the simulator, the performance of cosmos prototype is 16% lower

and 22.3% higher in read and program operations, respectively

Cosmos Prototype (Tiger2)

Architecture Exploration


Santa Clara, CA

23

Sequential Read Sequential Program

22.3% ─ 16%

C-Channel: Clustering-Channel Note: Cosmos prototype is used in this experiment

Simulator (FlashSim)

Cosmos Prototype

Simulator (FlashSim)

Cosmos Prototype

*Ref.) FlashSim: A Simulator for NAND Flash-Based Solid-State Drives

Th

rou

gh

pu

t (M

B/s

)

Th

rou

gh

pu

t (M

B/s

)

X-FTL

Transactional FTL for SQLite (SIGMOD 2013)

• Atomic write of data pages

– Offload the semantic of all-or-nothing propagation of data

pages carried out in host system down to FTL layer

– Avoid “redundant writes”

• Originally on top of Jasmine OpenSSD; now in porting to

Cosmos


Santa Clara, CA

24 *Ref.) X-FTL: Transactional FTL for SQLite Databases

X-FTL (Continued)

Architecture


Santa Clara, CA

25

-

File System

-

Update P1, P2, … , Pn, Commit/Abort

SQLite Application

P1

New copy of P1, … , Pn

Propagation at commit

TID

Transactional Page Mapping Table (X-L2P)

LPN PPNnew

: :

Ti P1

: :

Ti Pn

Page Mapping Table (L2P)

LPN PPN

: :

P1

: :

Pn

: :

Read(Pi), Write(Pi),

Commit(Ti), Abort(Ti) Storage Interface

P2 Pn P1 P2 Pn

Old copy of P1, … , Pn

X-FTL

Recovery

Commit/Abort

: :

Status

:

Active

:

Active

:

Write/ Read

File System Interface

Traditional FTL with Garbage Collection

Read(Ti, Pi) , Write (Ti, Pi),

fsync, ioctl(abort)

ActiveSSD

ActiveSort (HotStorage 2014)

• Accelerates external sorting

• Performs on-the-fly merge inside the SSD when

the results are requested

• Eliminates extra data transfer

• Increases the lifetime of SSDs


Santa Clara, CA

26

Active SSD

On-the fly processing

Data Result Next

processing

Request

*Ref.) Accelerating External Sorting via On-the-fly Data Merge in Active SSDs

Stay Tuned

Sources will be available soon at the

OpenSSD webpage

Sources could be updated by other users

except us (welcome all the time!)

And more activities will be posted to the

webpage as well


Santa Clara, CA

27

Call For Participation

Welcome any contributions from

• SSD manufacturers

• NAND flash vendors

• Research groups

• Individual developers

• …


Santa Clara, CA

28

Contributors


Santa Clara, CA

29

Youngin Jo

SSD Architecture

& PCIe Interface

Jaehyeong Jeong Ph.D.

System Architecture

& SSD HW Controller

Sanghyuk Jung Ph.D.

System Architecture

& SSD SW Firmware

Youngnam Kim

SSD HW Architecture

Sangjin Lee

Controller Architecture

& PCIe Interface

Taeyeong Huh

SSD HW Controller

Prof. Yong Ho Song Ph.D.

Hanyang University

Prof. Sang-Won Lee Ph.D.

Sungkyunkwan Univ

Prof. Jin-Soo Kim Ph.D.

Sungkyunkwan Univ

Sung-Rae Kim Ph.D.

ECC Algorithm

Gi-Hwan Oh

xFTL Design

Young-Sik Lee

Active SSD

Ilyong Jung

Controller Architecture

& ECC Engine

Luis Cavazos

Active SSD Gyeongyong Lee

SSD SW Firmware

Thank you For further information,

visit http://www.openssd-project.org


Santa Clara, CA

30

http://www.openssd-project.org/



cosmos openssd: a pcie-based open source ssd platform · pdf fileerror correction code bch 32...

Documents