computer structure x86 virtual memory and tlb

Computer Structure 2012 – VM1

Computer Structure

X86 Virtual Memory and TLB

Franck SalaSlides from Lihu and Adi’s Lecture


Virtual Memory

· Provides the illusion of a large memory

· Different machines have different amount of physical memory– Allows programs to run regardless of actual physical memory size

· The amount of memory consumed by each process is dynamic– Allow adding memory as needed

· Many processes can run on a single machine– Provide each process its own memory space– Prevents a process from accessing the memory of other processes

running on the same machine– Allows the sum of memory spaces of all process to be larger than physical

memory

· Basic terminology– Virtual Address Space: address space used by the programmer– Physical Address: actual physical memory address space


Virtual Memory: Basic Idea

· Divide memory (virtual and physical) into fixed size blocks– Pages in Virtual space, Frames in Physical space– Page size = Frame size– Page size is a power of 2: page size = 2k

· All pages in the virtual address space are contiguous

· Pages can be mapped into physical Frames in any order

· Some of the pages are in main memory (DRAM), some of the pages are on disk

· All programs are written using Virtual Memory Address Space

· The hardware does on-the-fly translation between virtual and physical address spaces

– Use a Page Table to translate between Virtual and Physical addresses


Virtual to Physical Address translation

Page size: 212 byte =4K byte

47

Page offset011

Virtual Page Number

11 0

Physical Page Number38

Virtual Address

Physical Address

V D Phy. page #

1

Page table basereg

0

Valid bit

Dirty bit

12

AC

Access Control- Memory type

(WB, WT, UC, WP …)- User / Supervisor

12Page offset


Translation Look aside Buffer (TLB)

· Page table resides in memory each translation requires an extra memory access

· TLB caches recently used PTEs – speed up translation

– typically 128 to 256 entries, 4 to 8 way associative

· TLB Indexing

· On A TLB miss– Page Miss Handler (HW PMH) gets PTE from memory

AccessPage TableIn memory

Yes

NoTLB Hit ?

Virtual Address

Physical Addresses

TLB Access

Tag Set

OffsetVirtual page number


Virtual Memory And Cache

TLB access is serial with cache access

Page table entries are cached in L1 D$, L2$ and L3 $ as data

Yes

Page Walk: get PTE from memory Hierarchy

No

Access Cache

Virtual Address

L1Cache Hit ?

Yes

No

Physical Addresses Data

No AccessMemory

L2Cache Hit ?

TLBHit ?

AccessTLB

STLBHit ?

No


Glossary· Page Fault

– Page is not in main memory– Context switch– Demand load policy

· Access control / Protection fault· Context switch· Page Aliasing

– DLL, Shared code, large Malloc· Page swap out

– snoop-invalidate for each byte in the cache (both L1 and L2) All data in the caches which belongs to that page is invalidated

The page in the disk is up-to-date

– If the TLB hits for the swapped-out page, TLB entry is invalidated

– In the page table The valid bit in the PTE entry of the swapped-out pages set to 0 All the rest of the bits in the PTE entry may be used by the operating system

for keeping the location of the page in the disk


32bit Mode: 4KB / 4MB Page Mapping· 2-level hierarchical mapping: Page Directories and Page tables

– 4KB aligned

· PDE– Present (0 = page fault)

– Page size (4KB or 4 MB)

CR4.PSE=1 both 4MB & 4KB pages supported

Separate TLBs

OFFSET

031

DIR TABLE

Linear Address Space (4K Page)

1121

1K entryPage Table1K entry

Page Directory

PDE

4K Page

data

CR3 (PDBR)

10 10 12

PTE

20+12=32 (4K aligned)

20

20

OFFSET031

DIR

Linear Address Space (4MB Page)

21

Page Directory

PDE

4MByte Page

data

CR3 (PDBR)

10 22

20+12=32 (4K aligned)

10


32bit Mode: PDE and PTE Format· 20 bit pointer to a 4K

Aligned address

· Virtual memory– Present– Accessed– Dirty (in PTE only)– Page size (in PDE only)– Global

· Protection– Writable (R#/W)– User / Supervisor #

2 levels/type only

· Caching– Page WT– Page Cache Disabled– PAT – PT Attribute Index

· 3 bits available for OS usage

Page Directory Entry (4KB page table)

Page Table Entry

UGPage Frame Address 31:12 AVAIL 0 A APCD

PWT

W P

PresentWritableUser / SupervisorWrite-ThroughCache DisableAccessedPage Size (0: 4 Kbyte)GlobalAvailable for OS Use

04 12357911 681231 -

GAVAILPage Frame Address 31:12 DPCD

PWT

U WA PPAT

PresentWritableUser / SupervisorWrite-ThroughCache DisableAccessedDirtyPATGlobalAvailable for OS Use

04 12357911 681231 -


4KB Page Mapping with PAEPhysical addresses is extended to M bits / Linear address remains 32 bit1995: Pentium Pro…

OFFSET

031

DIR TABLE


1121

1K entryPage Table1K entry

Page Directory

PDE

4K Page

data

CR3 (PDBR)

10 10 12

PTE

20+12=32 (4K aligned)

20

20

32 32

Dir ptr

029

DIR TABLE OFFSET


1120

512 entryPage Table

512 entryPage

Directory

PDE

4KBytePage

data

9 9 12

PTE

CR3 (PDPTR)

32 (32B aligned)

M-12

M-12

12213031

4 entryPage

DirectoryPointerTable

Dir ptr entryM-12

2

64

64


2MB Page Mapping with PAEPhysical addresses is extended to M bits / Linear address remains 32 bit

Dir ptr

029

DIR OFFSET


20

Page Directory

PDE

2MBytePage

data

9 21

CR3 (PDPTR)

32 (32B aligned)

M-21

213031

Page DirectoryPointerTable

Dir ptr entryM-12

2

64

64

OFFSET

031

DIR


21

Page Directory

PDE

4MByte

Page

data

CR3 (PDBR)

10 22

20+12=32 (4K aligned)

10

32

32


4KB Page Mapping in 64 bit Mode2003: AMD Opteron…

sign ext.

029

DIR TABLE OFFSET


1120

512 entryPage Table512 entry

Page Directory

PDE

4KBytePage

data

9 9 12

PTE

CR3 (PDPTR)

40 (4KB aligned)

M-12

M-12

12213038

512 entryPage


PDP entryM-12

9

PDPPML4

394763

512 entryPML4Table

PML4 entry

9

M-12

64

64 64 64

256 TB of virtual memory (248)1 TB of physical memory (240)

PML4: Page Map Level 4PDP: Page Directory Pointer


2MB Page Mapping in 64 bit Mode

PML4 PDPsign ext.

029

DIR OFFSET

Linear Address Space (2M Page)

20

512 entryPage

Directory

PDE

2MBytePage

data

9 21

CR3 (PDPTR)

40 (4KB aligned)

M-21

213038

512 entryPage


PDP entryM-12

9

394763

512 entryPML4Table

PML4 entry

9

M-12


1GB Page Mapping in 64 bit Mode

OFFSETPML4 PDPsign ext.

029

Linear Address Space (1G Page)

1GBytePage

data

30

CR3 (PDPTR)

40 (4KB aligned)

M-30

3038

512 entryPage


PDP entry

9

394763

512 entryPML4Table

PML4 entry

9

M-12


TLBs

· The processor saves most recently used PDEs and PTEs in TLBs– Separate TLB for data and instruction caches– Separate TLBs for 4KB and 2/4MB page sizes


Question 1· We have a core similar to X86

– 64 bit mode– Support Small Pages (PTE) and Large Pages (DIR)– Page table size in each hierarchy is the size of a small page– Entry size in the Page Table is 16 byte, in all the hierarchies

011N1 1263

sign ext. DIR TABLE OFFSETPDPPML4

N2N3N4

· What is the size of a small page ?

12 bits in the offset field 212 B = 4KB

· How many entries are in each Page Table?

Page Table size = Page Size = 4KBPTE = 16B 4KB / 16B = 212 / 24 = 28 = 256 entries in each Page Table


Question 1011N1 1263


N2N3N4

· What are the values of N1, N2, N3 and N4 ?

Since we have 256 entries in each table, we need 8 bits to address them- Table [19:12] N1 = 19- DIR [27:20] N2 = 27- PDP [35:28] N3 = 35- PML4 [43:36] N4 = 43

- 64 bit (large & small)- PT size = Page size = 4KB- PTE = 16B- Page Table: 256 entries

· What is the size of a large page ?

Large pages are pointed by DIRSo the large pages offset is 20 bits [19:0] large pages size: 220 = 1MB

We can also say: DIR can point to 256 pages of 4KB = 1MB


Question 101119 1263


273543

· We access a sequence of virtual addressesFor each address, what is the minimal number of tables that were added in all the hierarchies ?

· See next foil in presentation mode…

- 64 bit (large & small)- PT size = Page size = 4KB- Large page size = 1 MB- PTE = 16B- Page Table: 256 entries


PML4 PDP DIR PTE offset

027 1119

E2

4KBPage

FF

8 8 12

PageTable

CR3

12202835

PDPTable

82

8

364363

PML4Table

8


PageDir

Question 1: sequence of allocations

D328

B25

937

C00

68

49

71

46

B5

62254

00000 D3 82 FF E2 B25

00000 D3 82 FF E2 349

00000 D3 82 FF 68 937

349

00000 D3 82 28 49 C00

00000 71 46 B5 54 622

8 bits instead of 9 for example purposes only

3 new tables + 1 page

0 new table

0 new table + 1 page




Caches and Translation Structures

PlatformOn-dieCore

PMH

L1 data cacheL1 Inst. cache L2

Inst. TLB Data. TLB

translation

L3

Instruction bytes

PTEPTE

STLB

PTE

Memory

PDE cache

PDP cache

PML4 cache

PDE entry

PDP entry

PML4 entry

VA[47:12]

VA[47:21]

VA[47:30]

VA[47:39]

PageWalkLogic

Load entry


cache Accessed with virtual address bits

If hits, returns

PDE cache [47:21] PDE

PDP cache [47:30] PDP entry

PML4 cache [47:39] PML4 entry

Page Walk

SIGN EXT. PML4 PDP DIR TABLE OFFSET

01221303948


Question 2· Processor similar to X86 – 64 bits · Pages of 4KB· The processor has a TLB

– TLB Hit: we get the translation with no need to access the translation tables– TLB Miss: the processor has to do a Page Walk

· The hardware that does the Page Walk (PMH) contains a cache for each of the translation tables

· All Caches and TLB are empty on Reset

· For the sequence of memory access below, how many accesses are needed for the translations?

01119 1263


273543

Address memory access

Explanations

0000122334455666H

0000122334455777H

0000122884455777H







01119 1263


273543


Explanations

0000122334455666H

0000122334455777H

0000122884455777H







01119 1263


273543


Explanations

0000122334455666H

4 We need to access the memory for each of the 4 translation tables

0000122334455777H

0000122884455777H







01119 1263


273543


Explanations

0000122334455666H


0000122334455777H

0 Same pages as above TLB hit No memory access

0000122884455777H







01119 1263


273543


Explanations

0000122334455666H


0000122334455777H

0 Same pages as above TLB hit No memory access

0000122884455777H

3 We hit in PML4 cache. Then we miss in the PDP and so we need to access the memory 3 times: PDP, DIR, PTE


Question 2

· L1 data Cache: 32KB – 2 ways of 64B each· How can we access this cache before we get the physical

address?

64B 6 bits offset bits [5:0]32KB = 215 / (2 ways * 26 bytes) = 28 = 256 sets [13:6]

12 bits are not translated: [11:0] we lack 2 bits [13:12] to get the set addressSo we do a lookup using 2 un-translated bits for the set address

Those bits can be different from the PFN obtained after translation, therefore we need to compare the whole PFN to the tag stored in the Cache Tag array


Question 2: Read Acces

Offset

05

Set

613

Not Translated

12 11

VPN

Translated

47

40:12= =

Tag Match

Tag Match

Tag[40:12]

Tag[40:12]

Way 0 Way 1

PFN

40 13 12


Question 2:example

0000 0000 00000 … 00110000 0000 00000011 … 0000VB: PB:

Set: [13:6] = 0 Tag: 3 (0 if we don’t take [13:12])

Read Virtual Address B

0000 0000 00000 … 00000000 0000 00000000 … 0000VA: PA:

VPN Offset OffsetPFN

Set: [13:6] = 0

Write Virtual Address A

Tag: 0

Offset

05

Set

613

Not Translated

12 11

VPN

Translated

47


Question 2: Virtual Alias· L1 data Cache: 32KB · 2 ways of 64B each· What will happen when we access with a

given offset the virtual page A and after this, there is an access with the same offset in the virtual page B, which is mapped by the OS to the same physical page as A?

Offset

05

Set

613

Not Translated

12 11

VPN

Translated

47

PFN

40 13 12

VPNxxxx01

VPNyyyy00

PFNzzzz

- 2 virtual pages map to the same frame

xxxx01.set.ofset and yyyy00.set.ofset 01 and 00 are bits 13:12

PhysicalAddresses

Cache

Set[11:6]Not translated

Max: 64 sets

zzzz

Phys. addressed cache:The data exist only once

PhysicalAddresses

Cache

01.set[11:6] zzzz

zzzz00.set[11:6]

Virtual addressed cache:The data may exist twiceAVOID THIS !!!


Question 2: Virtual Alias· L1 data Cache: 32KB · 2 ways of 64B each· What will happen when we access with a

given offset the virtual page A and after this, there is an access with the same offset in the virtual page B, which is mapped by the OS to the same physical page as A?

Offset

05

Set

613

Not Translated

12 11

VPN

Translated

47

PFN

40 13 12

PhysicalAddresses

Cache

01.set[11:6] zzzz

zzzz00.set[11:6]


· Avoid having the same data twice in the cachexxxx01.set.ofset and yyyy00.set.ofset

· Check 4 sets when we allocate a new entry and see if the same tag appears

· If yes, evict the second occurrence of the data (the alias)


Question 2: Snoop· L1 data Cache: 32KB · 2 ways of 64B each· What happens in case of snoop in the

cache?

Offset

05

Set

613

Not Translated

12 11

VPN

Translated

47

PFN

40 13 12

PhysicalAddresses

Cache

01.set[11:6] zzzz

zzzz00.set[11:6]


The cache is snooped with a physical address [40:0]

Since the 2 MSB bits of the set address are virtual, a given physical address can map to 4 different sets in the cache (depending on the virtual page that is mapped to it)

So we must snoop 4 sets * 2 ways in the cache


Question 3· Core similar to X86 in 64 bit mode

· Supports small pages (pointed by PTE) and large pages (pointed by DIR).· Size of an entry in all the different page tables is 8 Bytes· PMH Caches at all the levels

– 4 entries direct mapped – Access time on hit: 2 cycles– Miss known after 1 cycle

· PMH caches are accessed at all the levels in parallel· In each level, when there is a HIT, the PMH cache provides the relevant

entry in the page table in the relevant level· In each level, when there is a miss: the core accesses the relevant page

table in the main memory.

· Access time to the main memory is 100 cycles, not including the time needed to get the PMH cache miss.

01123 1263


354755


Question 3

· What is the size of the large pages?

The large page is pointed by DIR, therefore, all the bits under are offset inside the large page: 224 = 16 MB

01123 1263


354755

· How many entries in each Page Table ?

• PTE: 212

• DIR: 212

• PDP: 212

• PML4: 28


01123 1263


354755

Virtual Addr. Cycles Comment

FF81 2345 6789 ABCD

FF81 2340 6789 ABCD

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD

TLB 4entries Direct mappedTLB hit: 2 cyclesTLB miss: 1 cycleMemory access: 100 cycle


01123 1263


354755


FF81 2345 6789 ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles

FF81 2340 6789 ABCD

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755



FF81 2340 6789 ABCD 202 (PML4, PDP, DIR, sTLB) = (H,H,M,M) 1cyc PDP TLB read 1 more cycle DIR TLB: Miss 100 cyclesPTE TLB: Miss 100 cycles

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755




FF80 2340 6789 ABCD 401 PMH cache miss in all the levels: 1 + 4*100 = 401 cycles

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755





FF81 2340 6709 ABCD 302 (PML4, PDP, DIR, sTLB) = (H,M,M,M) 1 cyc PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss2 + (3 × 100) = 302

FF81 2340 6709 A0CD



01123 1263


354755





FF81 2340 6709 ABCD 302 (PML4, PDP, DIR, sTLB) = (H,M,M,M) 1 cyc PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss2 + (3 × 100) = 302

FF81 2340 6709 A0CD 2 Hit in TLB – no need to go to PMH: 2 cycles



Backup Slides


Offset

05

Set

613

VPN

47

= =

Tag Match

Tag Match

Way 0 Way 3PFN

40 11

13

Tag field40 11:

Translated not translated

047 1112

Virtual Address

…


Offset

Set

VPN

= =

Tag Match

Tag Match

Way 0 Way 3PFN

40

Tag field :

Translated not translated

Virtual Address

…


Question 3

· We have a sequence of accesses to virtual addresses. For each address, how many cycles are needed to translate the address.

Assume that we use small pages only and that the TLBs are empty upon reset

01123 1263


354755



FF81 2345 6789 ABCD

FF81 2340 6789 ABCD

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD


01123 1263


354755


FF81 2345 6789 ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles

FF81 2340 6789 ABCD

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755



FF81 2340 6789 ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Miss 101 cyclesPTE TLB: Miss 101 cycles

FF80 2340 6789 ABCD

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755



FF81 2340 6789 ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Hit 101 cyclesPTE TLB: Hit 101 cycles

FF80 2340 6789 ABCD 404 TLB miss in all the levels: 4 *101 = 404 cycles

FF81 2340 6709 ABCD

FF81 2340 6709 A0CD



01123 1263


354755





FF81 2340 6709 ABCD 305 PML4 TLB: 2 cycles PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set missDIR & PTE: miss(1 × 2) + (3 × 101) = 305

FF81 2340 6709 A0CD



01123 1263


354755





FF81 2340 6709 ABCD 305 PML4 TLB: 2 cycles PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set missDIR & PTE: miss(1 × 2) + (3 × 101) = 305

FF81 2340 6709 A0CD 8 Hit in all TLBs:4 * 2 = 8


computer structure x86 virtual memory and tlb

Documents

page directories

invalidatedthe page

virtual space

memory spaceprevents

actual physical memory

virtual memoryprovides

main memory dram

physical address spacesuse