xpds16: virtual nvdimm in xen - haozhong zhang, intel
TRANSCRIPT
2
Outline
Introduction to NVDIMM
Overview of Virtual NVDIMM in Xen
Address Management
ACPI Emulation
Status & TODO
3
NVDIMM = Non-Volatile Dual In-line Memory Module
3
• Can be byte-addressable
• Persistent Storage
• retain data across power cycles
• Faster than SSD, higher density than RAM
44
CPU
RAM
NVDIMM
CPU
RAM NVDIMM
CPU
NVDIMMBLK Win
2-Level Memory
(2LM)
Persistent Memory
(pmem)
Persistent Block
(pblk)
✓
5
PMEM Architecture Overview
5
CPU
MC
RAM 0
RAM 1
SPA (System Physical Address):
/dev/pmemN
NVDIMM
Interleaveset
PMEM Namespace
ACPI NFIT
PMEM Region
PMEM Region
6
More about ACPI
6
CPU
MC
RAM 0
RAM 1 Label Storage Area
ACPI Namespace Device• Root “ACPI0012”
• _DSM: read/write label storage area
• _DSM: bad block scan (ARS)
• _DSM: vendor-specific
• _FIT: NVDIMM hotplug
• …
7
Cache Flush - ADR
7
CPU
MC
Cache
RAM 0
RAM 1
PMEM Region
PMEM Region
1. clwb/clflushopt/clflush
2. ADR (Asynchronous DRAM Refresh) automatically
8
Cache Flush – Flush Hint Structure
8
CPU
MC
Cache
RAM 0
RAM 1
PMEM Region
PMEM Region
1. clwb/clflushopt/clflush
2. Write to NFIT Flush Hint Structure
9
Outline
Introduction to NVDIMM
Overview of Virtual NVDIMM in Xen
Address Management
ACPI Emulation
Status & TODO
10
Usage Example of vNVDIMM in Xen
10
SPA
Hypervisor
Dom0
/dev/pmem0 /dev/pmem1
DAX file system
nvm1nvm2
/mnt/dax/
DomU
/dev/pmem0
DomU
/dev/pmem0
PMEM namespace 1 PMEM namespace 2
xl.cfg:
vnvdimm = [ ‘/dev/pmem0’ ]
xl.cfg:
vnvdimm = [ ‘/mnt/dax/nvm2’ ]
11
Architecture of vNVDIMM
11
vACPI
vDSM
vLabel
vACPI
MMIO
copy via hvmloader
QEMUxl:
• parse config
• get host pmem SPA
Linux Kernel (PMEM driver + DAX)
Dom0 DomU
Xen
Hypervisor
12
Outline
Introduction to NVDIMM
Overview of Virtual NVDIMM in Xen
Address Management
ACPI Emulation
Status & TODO
13
Detect Host PMEM Regions
13
• Dom0 PMEM Driver
• Parse NFIT and namespace labels
• Report PMEM SPA ranges to Xen hypervisor
• Xen Hypervisor
• Check overlap among PMEM SPA ranges and normal RAM regions
14
Guest Address Mapping
14
• Map as normal RAM
• EPT
• Fit into the existing memory management
• Reuse/Benefit from whatever already exists for normal memory management
• Room for Management Data Structures
• struct page_info per PMEM page
• Insufficient RAM space when PMEM is large
• Reserve and store on PMEM
15
Outline
Introduction to NVDIMM
Overview of Virtual NVDIMM in Xen
Address Management
ACPI Emulation
Status & TODO
16
hvmloader vs. QEMU
16
• Difficulties in Xen
• Not only dynamic data but also dynamic code in AML
• No runtime AML builder in Xen
• Conveniences in QEMU
• Runtime AML builder
• Existing NFIT and NVDIMM AML code implementation
17
Pass from QEMU
17
• Generate NFIT and AML code by QEMU
• Copy into DomU guest by QEMU
• Check and load by hvmloader
18
Collisions with hvmloader ACPI
18
• Duplicated tables
• Check table signature and reject duplicated tables (except SSDT)
• ACPI namespace device name collision
“NVDR”
Name (_HID, ”ACPI0012”)Method (_FIT) {…}Method (_DSM, …) {…}
+ Is “NVDR” already used?
Refuse to load
Device (NVDR) {Name (_HID, ”ACPI0012”)Method (_FIT) {…}Method (_DSM, …) {…}
}Yes
No
19
Outline
Introduction to NVDIMM
Overview of Virtual NVDIMM in Xen
Address Management
ACPI Emulation
Status & TODO
20
Status & TODO
20
• 2 RFC design documents
• https://lists.xen.org/archives/html/xen-devel/2016-02/msg00006.html
• https://lists.xen.org/archives/html/xen-devel/2016-07/msg01921.html
• Patches in development
• Xen / QEMU / Linux PMEM driver
• TODO
• NFIT Flush Hint Table
• RAS
• …
Q & A