ece 598-ms: advanced memory and storage systems

ECE 598-MS: Advanced Memory and Storage Systems

Jian Huang

Lecture 7: Unified Address Translation with

FlashMap

Use Flash As Non-Volatile Memory

2

HDD

(milliseconds)

SSD

(microseconds)

DRAM

(nanoseconds)

Application Memory

Component

Designed for

fast memory

Application Storage

Component

Designed for

slow disk

Application Memory

Component

Application Storage

Component


2

HDD

(milliseconds)

SSD

(microseconds)

DRAM

(nanoseconds)

Application Memory

Component

Designed for

fast memory

Application Storage

Component

Designed for

slow disk

Application Memory

Component

Application Storage

Component 0.3-0.9%40%


2

HDD

(milliseconds)

SSD

(microseconds)

DRAM

(nanoseconds)

Application Memory

Component

Designed for

fast memory

Application Storage

Component

Designed for

slow disk

Memory Mapped SSD

3

Application SSDVirtual Memory Filesystem

File

Memory Mapped SSD

3

Application SSDVirtual Memory Filesystem

File

mmap()

munmap()

msync()

Memory Mapped SSD

3

Application SSDVirtual Memory

Extended

Memory

YAY!

MORE MEM!!!

Filesystem

File

mmap()

munmap()

msync()

Memory Mapped SSD

3

Minimal Code

Modifications

+


Extended

Memory

YAY!

MORE MEM!!!

Filesystem

File

mmap()

munmap()

msync()

Memory Mapped SSD

3

Minimal Code

Modifications

+


+Data

DurabilityExtended

Memory

YAY!

MORE MEM!!!

Filesystem

File

mmap()

munmap()

msync()

No Free Lunch: Software Overhead

4

Virtual Memory System

Application

Flash Translation Layer

Flash

File System


4


Application


Flash

File System

Page Table &Memory Manager

Virtual Address

Physical Address/File Offset

page fault


4


Application


Flash

File System


Virtual Address


File Index

File Offset

Logical Block Address

page fault


4


Application


Flash

File System


Virtual Address


File Index

File Offset


FTL


Physical Block Address

page fault


4


Application


Flash

File System


Virtual Address


File Index

File Offset


FTL


Physical Block Address

page fault

3 address translations

+ 2 boundary checks

+ 2 permission checks

-----------------------------

= Latency:

15 – 20 microseconds

+ Metadata Overhead

FlashMap: Unified Address Translation

5

Application

Flash



File System

FlashMap: Unified Address Translation

5

Application

Flash

Unified Address Translation

1 Reduced Storage, only 1 mapping table

2 Reduced Latency

only 1 address translation + 1 permission check + 1 boundary check

FlashMapChallenges

6

Application


File System


Flash

FlashMapChallenges

6

Application


File System


Flash

Page table is process-specific & private

File index is system-wide shared resource

Mapping table is shared and should be persisted

Combining Page Table and File System

7


Flash

Process A

File SystemFile

Virtual Memory


7


Flash

Process A

File System

VM Region

File

Virtual Memory


7


Flash

Process A

File System

VM Region

Page Table

PGD PUD PMD PTE Offset

…

File

Virtual Memory


7


Flash

Process A

File System

VM Region

Page Table


…

File

Page table is process-specific & private

Virtual Memory


7


Flash

Process A

File System

VM Region

Page Table


…

File

Only for mapped file

Virtual Memory

Preserving File System Permissions

8


Flash

Process A

File System

VM Region

Page Table

Mapped File

Process B

VM Region

Page Table


…

Shared Page Table


8


Flash

Process A

File System

VM Region

Page Table

Mapped File

Process B

VM Region

Page Table


…

READ_ONLY READ_WRITE

Shared Page Table


8


Flash

Process A

File System

VM Region

Page Table

Mapped File

Process B

VM Region

Page Table


…


Permission

Conflict !!!

Shared Page Table


8


Flash

Process A

File System

VM Region

Page Table

Mapped File

Process B

VM Region

Page Table


…


Only share the leaf-level page table pagesShared Page Table


8


Flash

Process A

File System

VM Region

Page Table

Mapped File

Process B

VM Region

Page Table


…


Shared Page Table

Preserving Memory Protections

9


Flash

File SystemMapped File

Shared Leaf-level Page Table

…

Process A


Process B


Private

leaf-level page table

Private Private

Shared


9


Flash



…

Process A


Process B


Custom memory protection?Private



9


Flash



…

Process A


Process B


Private


Combining FTL and Shared Page Table

10


Flash



…

Process A


Process B



10Flash



…

Process A


Process B


Mapping Table Garbage Collection ECC Wear Leveling


10Flash



…

Process A


Process B


Garbage Collection ECC Wear Leveling

Overloaded PTE

Mapped File

Flash

Put It All Together

11

Process A

Shared Leaf-level Page Table Pages


Read

DRAM

…

Mapped File

Flash

Put It All Together

11

Process A



Read

DRAM

… page fault

Mapped File

Flash

Put It All Together

11

Process A



Read

DRAM

…update PTE

Mapped File

Flash

Put It All Together

11

Process A


PGD PUD PMD PTE OffsetDRAM

Write

…

Mapped File

Flash

Put It All Together

11

Process A



Write

…

DRAM hit

Mapped File

Flash

Put It All Together

11

Process A



Write

…

DRAM miss

Mapped File

Flash

Put It All Together

11

Process A



Write

…update PTE

Mapped File

Flash

Put It All Together

11

Process A



Write

…

Mapped File

Flash

Put It All Together

11

Process A



GC

…

Metadata Size for 1 TB SSD

12

0

1

2

3

4

5

6

7

Meta

dat

a Si

ze (

GB

)

Baseline (mmap + EXT4 + FTL with page level mapping)

FTL + FS (mmap + combined FTL & filesystem)

FlashMap

50%

Reduction

Latency Reduction with FlashMap

13

0

10

20

30

40

50

5 10 15 20 25

Lat

ency

(us)

Device latency (microseconds)

Baseline FTL+FS FlashMap

Latency Reduction with FlashMap

13

0

10

20

30

40

50

5 10 15 20 25

Lat

ency

(us)

Device latency (microseconds)


The combination of page table and file system

contributes mostly to the latency reduction

Benefits from Reduced Latency (Database)

14

0

5

10

15

20

25

25 20 15 10 5

Thro

ugh

put

(K T

PS)

Device Latency (us)


As Flash becomes faster,

FlashMap provides more performance benefits for latency-critical apps

FlashMapImpact

15

Metadata Size

50%

Performance

1.8x

Latency

53%

FlashMapImpact

15

Unifying emerging memory and storage

e.g., storage-class memory

Impact on industry

How to use scalable SSD in data center

ece 598-ms: advanced memory and storage systems

Documents