frederic vecoven sun microsystems - uliege.be
TRANSCRIPT
TAKE IT TO THE NTH
Frederic VecovenSun Microsystems
SunFire range of servers
System Components
Ultra−SPARCCPU &
compilers
Fireplane Shared
MemoryInterconnect
Applications &Middleware
Clustering & Networking
Operating Environment
Storage
Cache Coherency Basics
CPU 1 CPU 2 CPU 3
1. Read to Share
Memory
CPU 4
Coherency blocks(Aligned 32, 64, or 128 bytes)
2. Read to Share
4. Read to Share3. Read
to Own
5. Writeback
3. Invalidate 5. Invalidate
Cache Coherency Types
1. Broadcast (snoopy) coherency\\
P P P P\\
M M M MIO IO
\\
Snooping coherence domai n
PPPP\\
AgentMMMM
IOIO
P
M
IO
Processor
Memory
I/O controller
2. Point−to−point (directory) coherency
Glo
bal
Inte
rcon
nect
Snooping coherence domai n
Agent
Sun Generation Timeline
‘90 ‘95 ‘00 Now
UltraSPARC−I / UPA
SuperSPARC / XDBus
Cypress SPARC / MBus
UltraSPARC−III / Fireplane
Developmen t
Productio n
CPU core / Interconnect1
2
3
4
UltraSPARC−V5
Increasing CPU Integration
Memory controller
Cypress
IUCypress
FPU Cache tagsCache controller and Coherency controller
Ultra−IIIProcessor
Coherency controller
Memory controller
External cache
controller
External cache tags
Ultra−IProcessor
Coherency controller
Memory controller
External cache
controller
External cache tags
Coherency controller
Memory controller
SuperSparcProcessor
External cache controller and cache tags
Cypress SPARC / MBus — 1990
SuperSPARC / XDBus — 1993
UltraSPARC−I / UPA — 1996
UltraSPARC−III / Fireplane — 2000
1
2
3
4
Generation 1: MBus1
0.1 GBps4 CPUs
MBusCypress SPARC
1990
SnoopyCoherency
CPU
MEM
IO
1 bus
Generation 2: XDBus1 2
0.1 GBps4 CPUs
1.28 GBps64 CPUs
MBusCypress SPARC
1990
XDBusSuperSPARC
1993
SnoopyCoherency
SnoopyCoherency
CPU
MEM
IO
1, 2, or 4buses
1 bus
CPU
MEM
IO
Generation 3: UPA31 2
0.1 GBps4 CPUs
1.28 GBps64 CPUs
12.8 GBps64 CPUs
SnoopyCoherency
MBusCypress SPARC
1990
XDBusSuperSPARC
1993
UPAUltraSPARC−I/II
1996
SnoopyCoherency
SnoopyCoherency
CPU
MEM
IO
CPU
MEM
IOD
ata
Cro
ssba
ror
Data bus or xbar
1 or 4address buses
1, 2, or 4buses
1 bus
CPU
MEM
IO
31 42
Generation 4: Fireplane
0.1 GBps4 CPUs
1.28 GBps64 CPUs
43 /172 GBps72–106 CPUs
12.8 GBps64 CPUs
DirectoryCoherency
SnoopyCoherency
MBusCypressSPARC
1993
XDBusSuperSPARC
1993
UPAUltraSPARC−I/II
1996
SnoopyCoherency
SnoopyCoherency
CPU
MEM
IO
Dat
a C
ross
bar
FireplaneUltraSPARC−III
2000
CPU
MEM
IO
Add
ress
Cro
ssba
r
Dat
a C
ross
bar
CPUMem
IO
CPUMem
IO
CPUMem
IO
Res
pons
e C
ross
bar
Snoopy
9.6 GBps
or
Data bus or xbar
1 or 4address buses
1, 2, or 4buses
1 bus
CPU
MEM
IO
Worldwide Unix Server Market
1997 1998 1999 2000 2001$0
$5
$10
$15
$20
$25
$30
Fac
tory
Rev
enue
(B
illio
ns)
YearYears are 4 quarters endin g June 30Source: IDC Sept 01
System CPU capacity:
17−32
5−8
1−2
3−4
9–16
33−6465–128>128
32%
30%
38%
Solaris on SPARC on Fireplane Interconnect
Sun Fire Servers
280R
• 2 CPUs
• 8 GB RAM
• 4 PCI slots
V880
• 8 CPUs
• 32 GB RAM
• 9 PCI slots
3800−6800
• 8– 24 CPUs
• 64–192 GB RAM
• 16–32 PCI or12–16 cPCI slots
• 1 to 4 domains
15K
• 72–106 CPUs
• 576 GB RAM
• Up to 72 hot−swap PCI slots
• 1 to 18 domains
1−level Fireplane: Small Server
Workstation or Small server
• CPU/Memory pair
• PCI bridge
CPU Data Switch
CPU
CPU
PCIbridge
33 MHz slot(s)
66 MHz slot
Memory
Memory2.4 GBps2.4 GBps
2.4 GBps2.4 GBps
1.2 GBps
Level 0
2−level Fireplane: Workgroup
CPU Data Switch
CPU Memory
CPU
CPU Data Switch
CPU
CPU
Address Repeater
6x6Data
Switch
PCIbridge 66 MHz slot
PCIbridge 66 MHz slot
Memory
Memory
Memory2.4 GBps2.4 GBps4.8 GBps
2.4 GBps2.4 GBps
1.2 GBps
Workgroup server
• 4 Dual−CPU/ Memory boards
• 2 PCI bridges
Level 1 Level 0
33 MHz slot(s)
33 MHz slot(s)
3−level Fireplane: Mid−size
CPU Data Switch
CPU Memory
CPU
CPU Data Switch
CPU
CPU
Address Repeater
Data Switch
PCIbridge 66 MHz slot
PCIbridge
Data Switch
10x10Data
Switch66 MHz slot
Memory
Memory
Memory
Address Repeater
10−wayAddress Repeater
2.4 GBps2.4 GBps4.8 GBps4.
8 G
Bps
2.4 GBps2.4 GBps
2.4 GBps 1.2 GBps
4 Fireplane Switch Boards
I/O Assembly
6 Uniboards
4 I/O boards
CPU/Mem Uniboard
Level 2 Level 1 Level 0
33 MHz slot(s)
33 MHz slot(s)
Level 2 Level 1 Level 04−level Fireplane: Large Server
CPU Data Switch
CPU Memory
CPU
CPU Data Switch
CPU
CPU
Address Repeater
Data Switch
PCIbridge 66 MHz slot
PCIbridge
Data Switch
Data Switch
18x18 Data Xbar
18x18 Response
Xbar
18x18 Address
Xbar
66 MHz slot
Memory
Memory
Memory
Address Repeater
Address Repeater
2.4 GBps2.4 GBps4.8 GBps4.
8 G
Bps
2.4 GBps2.4 GBps
2.4 GBps4.8 GBps 1.2 GBps
Level 3
Expander Board
18 B
oard
sets
I/O Assembly
Centerplane
33 MHz slot(s)
33 MHz slot(s)
CPU/Mem Uniboard
CPU/Memory Uniboar d
Four banks of 8 SDRAM DIMMs
4 Data Switch ASICs
DataControl ASIC
AddressASIC
Boot bus ASIC
Boot bus
ASIC
Two sets of 8 CPU Data Switch ASICs
19.35"
16.5"C
PU
E$ DIMMs
CP
U
CP
U
CP
U
Pow
erP
ower
Pow
er
I/O Assemblies4 slot cPCI 6 slot cPCI
8 slot PCI 4 slot hot−swap PCI
Mid Range CabinetsSun Fire 68003 x Sun Fire 3800
3 x (8 CPUs + 12cPCI slots ) 24 CPUs + 32 PCI slot s
Sun Fire 4800/4810
12 CPUs + 16 PCI slot s
High−end Cabinet4 Fan Trays
18 Boardsets with18 CPU−Memory Boards
and 18 I/O orDual CPU Boards
4 Fan Trays
2 System Controllers
Six Dual Input 4 KWAC to 48 volt DCPower Supplies
75"
33"65"
SF 15K Components
CPU−Memory board (18)
I/O or MaxCPU board (18)
System Controller board (2)
System Controller peripheral board (2)
Control expander frame (2)
System expander frame (18)
Expander board (18)
Fan trays (8)
Fan Center−planes (8)
Power centerplane
Logic centerplane
Control expander sockets (2)
System expander sockets (18)
Centerplane ASICs (20)
(One side shown )
Mid−range Micro BenchmarkParallel pointer−chasin gUPA to Fireplane Generation
250
350
450
550
650
0 4 8 12 16 20 24
Mem
ory
late
ncy
(ns)
Processors
Sun Fire 6800(Switch)
SunEnterprise6500 (Bus)
(Loweris better)
0
1
2
3
4
5
6
0 4 8 12 16 20 24
Mem
ory
band
wid
th (
GB
ps)
Processors
(Higheris better)
SunEnterprise
Linea
r
6500 (Bus)
Sun Fire 6800(Switch)
Sun Interconnect Generations
MBus XDBus UPA Fireplane
Year (in mid−size servers) 1991 1993 1996 2001
System clock (MHz) 40 50–55 83–100 150
Coherency type
Packet switching
Address & Data
Coherency block (bytes)
Clocks/snoop
Address bus BW (GBps)
# Address buses
Datapath width (bytes)
Wiring
Broadcast Broadcast & point−to−point
Circuit Packet switched
Together Separate
32 64
16 11 2 1
0.08 0.3 3.0 9.6
1 4 4 18
8 16 32
Bused Switched
Max data B/W (GBps) 0.08 1.3 12.8 172
Mid: BusedHigh: Switched
31 42
Sun Snooping Bandwidth Progress
0.04
0.1
1
10
40
1990 1992 1994 1996 1998 2000 2002 2004
Bro
adca
st−b
us
band
wid
th (
GB
ps)
Year of first shipment in medium−sized servers
1. MBus
2. XDBus
3. UPA
Doubling every 18 months trend line
4. Fireplane
Now 5. US−V bus
Snoopy Bandwidth Progress
0.1
1
10
US−V
XDBus• 50 MHz (11 clock snoop)
• Packet switched
• 64−byte cache line
• Multiple buses
UPA
Fireplane
MBus• 40 MHz
Five Sun SMPGenerations
.05
1990 1993 1996 2000
Sno
opy
band
wid
th (
GB
ps)
1
2
3
4
• 100 MHz
• 2−clock snoop
• Separate address & data
• Routers & domains
• 150 MHz
• 1−clock snoop
• 2−level coherency
5
2003