a smart port card tutorial --- software john dehart washington university [email protected]
DESCRIPTION
A Smart Port Card Tutorial --- Software John DeHart Washington University [email protected] http://www.arl.wustl.edu/~jdd. Software Overview. SPC Specific Kernel Modifications Building a Kernel config compile Loading Kernel onto SPC to execute it download via APIC - PowerPoint PPT PresentationTRANSCRIPT
January 10,11 2000 Kits Workshop 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
A Smart Port Card
Tutorial---
Software
John DeHartWashington University
[email protected]://www.arl.wustl.edu/~jdd
January 10,11 2000 Kits Workshop 2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Software Overview
• SPC Specific Kernel Modifications• Building a Kernel
– config
– compile
• Loading Kernel onto SPC to execute it– download via APIC
• Initiating Kernel execution– boot loader
• Configuring Devices– APIC
January 10,11 2000 Kits Workshop 3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC Kernel Modifications
• Kernel Configuration File:– Memory Disk
– Serial Console
– Most device drivers unneeded (e.g no Ethernet…)
• Source Code Changes:– PCI Device Configuration: ./dev/pci/if_apic_pci.c
• APIC Specific configuration handled by driver
– Change some APIC defaults: ./dev/ic/apic.c• made switch port the default
– No HLT instruction: locore.s
– Kernel Memory Page Allocation: • ./arch/i386/include/pmap.new.h
– RTC: ./arch/i386/isa/clock.c, ./ufs/ffs/ffs_vfsops.c• We still had to remove some RTC code...
January 10,11 2000 Kits Workshop 4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Steps to an Executing Kernel on an SPC
• Configure Kernel– Start with a typical NetBSD kernel config file
– Add SPC specific option:• options SPC
– add memory disk (filesystem will reside in kernel!)• options MEMORY_DISK*
• options MINIROOTSIZE
– serial console port• options CONSDEVNAME
• options CONS_OVERRIDE
– remove “extra” devices (Ethernet, mouse, …)
January 10,11 2000 Kits Workshop 5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Steps to an Executing Kernel on an SPC
• Filesystem– we use a vnode disk device: vnd(4)
• disk device in a regular file
• pseudo-device vnd– must be included in your kernel where you are building the FS
– Does NOT need to be included in your kernel for the SPC
– build filesystem• dd(1): create the file
• vnconfig(8) - configure vnode pseudo disk as a regular file
• disklabel(8) - label it
• newfs(8) - construct a new file system
• mount(8)/umount(8): mount/unmount filesystem
– mknod(8) - make device files
– populate regular files from an existing system
January 10,11 2000 Kits Workshop 6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
… Steps to an Executing Kernel on an SPC
• Compile Kernel• Make a symbol only version of kernel for /netbsd
– ps, netstat, … use symbols from /netbsd
– real kernel with file system CANNOT reside in itself!
• Add File System to Kernel– mdsetimage(8)
• Create an image version of Kernel– sets up bss section and symbol table
• Kernel bss file is ready to download!
January 10,11 2000 Kits Workshop 7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
… Steps to an Executing Kernel on an SPC
• Reset SPC• Download Boot Loader
– via APIC control cells – loaded at physical memory location: 0x0
• Flip bit at location 0xFFE00 to start boot loader– configure NorthBridge (before kernel download!!)
• cache, memory size, memory type, timing, etc
– wait for kernel download to complete• looks for location 0x3FC to change from 0 to 1
– set up parameters on stack and jump to kernel• kernel located at 0x100000
• Download kernel bss file• Flip bit at location 0x3FC - kernel starts booting...
January 10,11 2000 Kits Workshop 8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
The Download Process
0x0 Bootloader
0x0100000Kernel
0 0 0 0 0 0 0 0
0x0FFE00
0x03FC
Boot ROM0xFFFFFFF0
0 0 0 0 0 0 0 1
0 0 0 0 0 0 0 00 0 0 0 0 0 0 1
RESET
DownloadProgram
January 10,11 2000 Kits Workshop 9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Host APIC to SPC Control Connections
APIC
PC
WUGS Switch
Port 0Port 1
Port 7
Port 6
Port 3Port 5
SPC/Port 4
Port 2
0/42
0/41
0/33 (0x21)
0/0x0321APIC ID
41 0x0321
Example
January 10,11 2000 Kits Workshop 10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
APIC Driver at SPC Boot Time
• PCI Configuration Space (sys/dev/pci/if_apic_pci.c):
– Base memory address:• APIC Physical memory in SPC starts at 0xF0000000
– Memory mapping options:• Memory Enable• Master Enable• Write Invalidate Enable
– IRQ• Driver normally would read this from PCI config space• APIC statically assigned IRQ 5 by System FPGA• For consistency, driver writes 5 to Interrupt Line Register
January 10,11 2000 Kits Workshop 11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
PCI Configuration Space
Configuration Space Header
Device ID Vendor IDStatus Command
Class Code Revision ID
BIST HeaderType
Cardbus CIS PointerSubsystem ID Subsystem Vendor ID
Expansion ROM Base AddressReservedReserved
Max_Lat Min_Gnt
LatencyTimer
Cache LineSize
Base Address Registers
InterruptPin
InterruptLine
January 10,11 2000 Kits Workshop 12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Defining a Route on APIC Network Interface• Configure the APIC
> ifconfig apic0 inet 192.168.10.1 netmask 0xffffff00- network interface: apic0- our IP address: 192.168.10.1
• Configure a VCI> atm_ifconfig apic0 0xc5 open -aal5 -llc -besteffort
-maxqueuebufs 100 -inport 0 -outports 1- network interface: apic0- vci: 0xc5- besteffort (vs. paced or lowdelay)- Tx buffer queue limit for this VCI: ‘maxqueuebufs 100’
• Configure the route> route add -iface 192.168.10.5 -link apic0:0.0.0.c5
- destination: 192.168.10.5- network interface: apic0- vci: 0xc5
January 10,11 2000 Kits Workshop 13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
atm_ifconfig> atm_ifconfigUsage: atm_ifconfig <interface> <vpivci> [open|close|modify|getall [options...]]
Options are:
-aal0 | -aal5
-llc | -nollc <protocol>: use LLC/SNAP?
protocol is inet | ratm | value of ethernet type
-loopback
-inport <input_port_number>
-outports <bitvector_of_output_ports>
bitvector: 1 => port 0, 2 => port 1, 3 => both
-lowdelay | -paced <rate> | -besteffort
-maxqueuebufs <max_descriptors_in_tx_queue>
<rate> is in Kbits/sec
January 10,11 2000 Kits Workshop 14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
SPC
TI
Switch Fabric
IPP
OP
P
SPC
TI
IPP
OP
P
TI
ControlProcessor
APIC Configuration
6 56 5 6 5
IPP
OP
P
SPC
TI
IPP
OP
P
SPC
TI
. . .
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
33 3 66 6
SPC
January 10,11 2000 Kits Workshop 15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Inp
ort
1
Fiber
Ou
tpo
rt 2
Ou
tpo
rt 1
Inp
ort
0
Defining APIC Ports for Routing
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
SwitchINPORT OUTPORTs
0 1
0 2
1 1
1 2
0 3
1 3
January 10,11 2000 Kits Workshop 16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Caveats
• You cannot change the IRQ assignments:– They are fixed in the System FPGA
– And there should be no reason to change them
• HLT instruction in locore.s commented out• PCI bus may be 25MHz or 33 MHz
– Adjustment of NetBSD clock may be needed
• Memory/Filesystem tradeoffs– we have not pushed the limits yet
• Understand APIC port numbering• SPC needs to be reset before EACH download