porting netbsd to the latticemico32 open source cpu by yann sionneau
TRANSCRIPT
About me
• Yann Sionneau • Embedded soGware developer • Working for M-‐Labs – hOp://m-‐labs.hk
• @yannsionneau on twiOer • Email: ys@m-‐labs.hk
Agenda
• I) The hardware part: the MMU – How La5ceMico32 MMU works
• II) The soGware part – My journey por$ng NetBSD to a new CPU
La5ceMico32 CPU
• 32 bits Harvard Architecture RISC • Big Endian • 6 stages • Fully bypassed • Op$onal configurable I/D caches – Direct mapped or – 2-‐way set associa$ve
• Wishbone on-‐chip bus
La5ceMico32 , Good points
• Small • Portable (works with several FPGA vendors) • Fast (~100 MHz on Slowtanpartan 6, 125 MHz on Kintex-‐7, up to 200 MHz without DDR3)
• Actually works • GCC/Binu$ls/GDB/Qemu/uCLinux/OpenWRT support
• OPEN SOURCE
Used in…
• Closed source commercial ASICs • Open source projects
• Can achieve 800 MHz in TSMC 90nm standard cell process
What’s the MMU’s job?
• Translate « virtual addresses » into « physical addresses »
• Memory protecGon against unwanted execu$on of code or data write (e.g. soGware bug or security issue) – Memory right access management
Main Memory
CPU pipeline VA PA
VA : Virtual Address PA : Physical Address
How does the MMU know the VA-‐>PA transla$on ?
Why « Page »?
• 0x00000004 -‐> 0x10000000 • 0x00000005 -‐> 0x10000001 • 0x00000006 -‐> 0x10000002 Etc…
Why « Page »?
• 0x00000004 -‐> 0x10000000 • 0x00000005 -‐> 0x10000001 • 0x00000006 -‐> 0x10000002 Etc…
This is WRONG!!!
Why « Page »?
• 0x00000*** -‐> 0x10000*** • 0x00001*** -‐> 0x10001*** • 0x00002*** -‐> 0x10002*** Etc…
Main Memory
CPU pipeline VA PA
VA : Virtual Address PA : Physical Address
Page Table
TLB TLB : Transla$on Lookaside Buffer
Main Memory
CPU pipeline VA PA
VA : Virtual Address PA : Physical Address
Page Table
TLB OperaGng System
Updates the
Gets informa$on from the
Updates the
Features?
• Page size – Only 4 kB
32 bits physical address : xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx
How many bits of an address indicate the offset within a given page?
Features?
• Page size – Only 4 kB
32 bits physical address : xxxxxxxx xxxxxxxx xxxx xxxx xxxxxxxx
Page number [31:12]
20 bits
Offset [11:0]
12 bits
Features?
• 2 TLB (Transla$on Lookaside Buffer) – ITLB – DTLB
• Each TLB contains 1024 entries – How many bits needed to index the TLB?
Features?
• 2 TLB (Transla$on Lookaside Buffer) – ITLB – DTLB
• Each TLB contains 1024 entries – How many bits needed to index the TLB?
10 bits!
Virtual address
Load or store?
Instruc$on or Data?
Physical address
Access granted/denied
I don’t know!
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001004
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004
Page number
Offset in the page
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001
VPN = 0xA0001 à 1010 0000 0000 0000 0001
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001 TLB index = 1
VPN = 0xA0001 à 1010 0000 00 00 0000 0001
TLB index, used to select a TLB line
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001 TLB index = 1
VPN = 0xA0001 à 1010 0000 00 00 0000 0001
TLB index, used to select a TLB line
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001 TLB index = 1
VPN = 0xA0001 à 1010 0000 00 00 0000 0001
Tag = 0x280 à 1010 0000 00 =
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001 TLB index = 1
VPN = 0xA0001 à 1010 0000 00 00 0000 0001
Tag = 0x280 à 1010 0000 00 =
⇒ Physical page number = 0xB0001
Line index Tag [10] Physical page number [20] Read-‐only [1]
Valid [1]
0 0xABC 0xABC00 0 0
1 0x280 0xB0001 1 1
2 0x300 0x00001 0 1
The TLB VA = 0xA0001 004 Page offset = 4 Virtual Page number = 0xA0001 TLB index = 1
VPN = 0xA0001 à 1010 0000 00 00 0000 0001
Tag = 0x280 à 1010 0000 00 =
⇒ Physical page number = 0xB0001 ⇒ Physical Address = 0xB0001004
Por$ng NetBSD
• 1°) NetBSD cross compila$on toolchain – build.sh – Makefiles here and there – Arch-‐specific directories
Allows to do: $ ./build.sh -‐U -‐m milkymist tools
Por$ng NetBSD
• 2°) Support for built-‐ins in libkern – NetBSD kernel is
• Not linked against libgcc • Linked against libkern
– Need to implement basic arithme$c func$ons emiOed by gcc in object code
– Implementa$on in sys/lib/libkern/arch/lm32
Por$ng NetBSD
• 3°) Building my first kernel – Create sys/arch/lm32 and sys/arch/milkymist – Populate
• sys/arch/<cpu|soc>/include • sys/arch/<cpu|soc>/conf
– Stub, stub, stub… Allows to do: $ ./build.sh -‐m milkymist -‐U kernel=GENERIC
Por$ng NetBSD
• 4°) Write basic console driver for early prints
struct consdev milkymist_com_cons = { […] milkymist_com_cngetc, /* cn_getc: kernel getchar interface */ milkymist_com_cnputc, /* cn_putc: kernel putchar interface */ […] };
Por$ng NetBSD
• 5°) Implement excep$on handlers • 6°) Call milkymist_startup() C code – Ini$alize console driver
• -‐> consinit() -‐> milkymist_uart_cnaOach() • cn_tab = &milkymist_com_cons;
– Ini$aliaze virtual memory subsystem • Call MD pmap_bootstrap()
– Let the kernel boot • Call NetBSD MI main()
Por$ng NetBSD
• 7°) Implement pmap.9 pmap -‐-‐ machine-‐dependent por/on of the virtual memory system – pmap_bootstrap() – pmap_init, pmap_create, pmap_destroy … – SW managed TLB? -‐> sys/uvm/pmap/
– used in (PowerPC Booke and LM32)
Por$ng NetBSD
• 8°) Implement copyin/copyout • 9°) Implement atomic opera$ons – No atomic instruc$on à RAS (Restartable Atomic Sequence) CAS (Compare And Swap)
– Other atomic and locking ops built around this CAS
RAS CAS int _atomic_cas_32(vola$le uint32_t *val, uint32_t old, uint32_t new); _atomic_cas_32: _atomic_cas_ras_start:
lw r4, (r1+0) /* load *val into r4 */ bne r4, r2, 1f /* compare r4 (*val) and old (r2) */ sw (r1+0), r3
_atomic_cas_ras_end: 1:
mv r1, r4 /* return (*val) */ ret
Por$ng NetBSD
• 10°) Add support for interrupts – Write a func$on to register interrupt handlers
• 11°) Have a running system clock – Write cpu_initclocks() – Write clock irq handler
• Call hardclock()
Other func$ons to write
• Switch context from one thread to another – cpu_switchto(9)
• Copy data and abort on page fault – kcopy(9)
• Save current context – sebault()
• Low level code to finish up fork() opera$on – cpu_lwp_fork(9)
Other func$ons to write
• Block interrupts to protect cri$cal sec$ons – spl(9)
• Init CPU and print copyright message – cpu_startup(9)
• Determine the root file system device – cpu_rootconf(9)
• Etc…
Por$ng NetBSD
• To boot user space – Create dummy ramdisk with /sbin/init – Build kernel with MFS – Insert ramdisk with mdse$mage – Boot it!
NetBSD/milkymist virt. Memory Layout
Kernel space User space
0 0xffffffff
0xc0000000
0xc8000000 RAM window
User stack
Kernel stack
DDR SDRAM : 128 MB
RAM window
Physical RAM
128 MB
Start: PA 0x4000 0000
End: PA 0x4800 0000
Virtual Memory Layout
VA: 0xc000 0000
VA: 0xc800 0000
Direct mapping
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x1234 1000
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x1234 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Binary à 0000000011 0000000010 101010111100 10 bits 10 bits 12 bits
1024 entries 1024 entries
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x1234 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Binary à 0000000011 0000000010 101010111100
1024 entries 1024 entries
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x1234 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Binary à 0000000011 0000000010 101010111100
1024 entries 1024 entries
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x4123 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Binary à 0000000011 0000000010 101010111100
1024 entries 1024 entries
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x4123 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2ABC
Binary à 0000000011 0000000010 101010111100
Physical address: 0x4123 1xxx
1024 entries 1024 entries
Page table
Page directory @ 0xc300 0000 0x**** ****
0x**** ****
0x**** ****
0xc400 1000
0x**** ****
Page table @ 0xc400 1000 0x**** ****
0x**** ****
0x4123 1xxx
0x**** ****
0x**** ****
Virtual address: 0x00C0 2 ABC
Binary à 0000000011 0000000010 101010111100
Physical address: 0x4123 1ABC
1024 entries 1024 entries
Accessing the page table
• Kernel running with MMU ON o Direct access through virt. addresses 0xc*** ****
• TLB miss handler running with MMU OFF o Computes phys. addresses from RAM window addresses using following formula
o PA = VA – 0xC000 0000 + 0x4000 0000
Follow progress
• hOp://sionneau.net/edgebsd/ • hOp://edgebsd.org/edgebsd/wiki/14/port-‐lm32
• hOp://github.com/fallen/NetBSD • hOp://git.edgebsd.org/gitweb/?p=edgebsd-‐src.git;a=shortlog;h=refs/heads/fallen-‐port-‐milkymist (aka hOp://goo.gl/9Mbg7U)
Thank you!
Sébas$en Bourdeauducq, Michael Walle, Robert Swindells, Stefan Kris$ansson, Lars-‐Peter Clausen, Pierre Pronchery, Radoslaw Kujawa, Youri Mouton, MaO Thomas, tech-‐kern@, M-‐Labs mailing list, and many more.