Toshiba

A Physical A Physical PerspectivePerspective of of Computer ArchitectureComputer Architecture

Peter Hsu, Ph.D.Peter Hsu, Ph.D.Chief ArchitectChief Architect

Microprocessor DevelopmentToshiba America Electronics Components, Inc.

Presented February 13, 2001 at Univ. of Wisconsin in Madison

A Physical View of Computer Architecture 2

IntroductionIntroduction

Computer Architecture Perspectives– Logical, Performance

• Abstract• Quantitative• Academia

– Physical, Cost• Constrained by History, Emotions, Physics• Modulated by Current World Affairs• Apprenticeship

A Physical View of Computer Architecture 3

ContentContent

Tour a Particular Design Point– Rationale of Choices– Confluence of Decisions– Rules of Thumb

Computer Markets– PC Infrastructure Defines Most Cost-Efficient– Consumer Volume Advancing Technology– All other Computers Competing with PC

A Physical View of Computer Architecture 4

Multichip ModuleMultichip Module

88 chip stacks

silicon substrate

printed circuit board

pressure plate 3000 wire bonds

alignment cage

heat distributor

12mm

4mm

14cm

10mm

A Physical View of Computer Architecture 5

Chip StackChip Stack

DRAMs

processors

router

12mm

10mm

0.3mm

12mm

10m width 20m pitch

stack shown upside down

A Physical View of Computer Architecture 6

Stack to Substrate ConnectionStack to Substrate Connection

silicon substrate

router chip

DRAMs

wirebondsprings

conventionalwirebond pads

heat

A Physical View of Computer Architecture 7

19 inches

1.5 in

flex signal PCB

rigid power PCB

heat pipe

multichipmodule

heat sink

System UnitSystem Unit

input/output connectors

A Physical View of Computer Architecture 8

Scalable ConfigurationsScalable Configurationsperipherals system units peripherals

office (110V 15A)copier room (220V 30A)

64-node supercomputer (80 Kilowatt)

A Physical View of Computer Architecture 9

Physical ArchitecturePhysical Architecture

cables between system units

system unit

silicon multichip substrate

chip stack

CPU

router

DRAMs

chip stack

CPU

router

DRAMs

chip stack

CPU

router

DRAMs

A Physical View of Computer Architecture 10

Logical ArchitectureLogical Architecture

byte-wide point-to-point networksystem unit

chip stack

level 2cache

mainmemoryrouter

L1$

CPU

L1$

CPU

L1$

CPU

L1$

CPU

serial point-to-point cable networkserial point-to-point cable network

A Physical View of Computer Architecture 11

Guiding PrinciplesGuiding Principles

Performance1. Latency (Memory, Interprocessor, etc.),2. Bandwidth, then

3. Microarchitecture

Cost– Silicon Portion Scales With Process

(e.g. Learning curve of copper-on-silicon substrate)

– Non-Silicon Portion Does Not Scale(e.g. Liquid immersion cooling hardware)

A Physical View of Computer Architecture 12

Silicon SubstrateSilicon Substrate

12mm chip

12mm

4mm spacer

4mm

maximum cutset2048 p-to-p links

150m pitch

3200 wire bondssubstrate to PCB

200mm(8 inch)wafer

maximum tracelength 24.8cm

14cm

A Physical View of Computer Architecture 13

Stack to Substrate ConnectionStack to Substrate Connection

chipstack 250m pitch

125m pad

125m space

75m clearance(0.003 inch or 3 mils)

75m tolerance

alignment cage

chip stack

substrate

Rule of thumb• Machined parts need several mils tolerance

A Physical View of Computer Architecture 14

Substrate DesignSubstrate Design

Internal Signals– Link

• 8 data, 2 clock bits (20% overhead)

• Source Synchronous

– Density• 20,480 signals across cutset ( 7m per track)

• 63210 1260 signals / stack (2304 total)

Rule of thumb• High speed 50% signal pads

A Physical View of Computer Architecture 15

Substrate Design (con’t)Substrate Design (con’t)

External Connections– Signal

• Node to multicomputer node (2in 2out 64)

• Node to peripheral device (2in 2out 64)

– Power• 1280 power/ground pairs• 20W per stack (VDD 1V)

Rule of thumb• A wire bond 1A sustained current

A Physical View of Computer Architecture 16

Interconnect DimensionsInterconnect Dimensions

4 3.5 7.5

6

5 10 5.5 15 7.5

8 VDD

VSS

2 3

width W space S pitch

height H

insulation thickness T

4.5

A Physical View of Computer Architecture 17

Electrical CharacteristicsElectrical Characteristics

[+ 0.06 + 1.66 0.14 ]( )W

T ( )H

T ( )H

T

0.222

( )T

S

1.34

= 1.15 + 2.80( )W

T

0.222

C

( )H

T

0.222

R = L

W HZ0 =

C0 C

Bakoglu, H.B., Circuits, Interconnections, and Packaging for VLSI, Addison-Wesley, 1990.

A Physical View of Computer Architecture 18

Lossy Transmission LineLossy Transmission Line

1

V

time0

1

V

time0

1

V

time0

Self terminating if Z0 R 2Z0

A Physical View of Computer Architecture 19

Substrate Design (con’t)Substrate Design (con’t)

Construction– Material

• Copper, 1.7 mcm• “Low-k” Insulator, 3.0

– Design Rules• 1 L 7.2cm R 51 Z0 27

• 2 L 18.4cm R 52 Z0 26

• 3 L 24.8cm R 47 Z0 27

– 7 Layers (3 X•Y pad)

A Physical View of Computer Architecture 20

Package DesignPackage Design

Thermal– System Unit

• Ambient: A 40C

• Airflow: 1 m/s (200 ft/min)• 1280W 16 16 1in

Rule of thumb• Heat sink with fan dissipates 5W per inch3

A Physical View of Computer Architecture 21

heat pipe

boilcondense

metal wick

gas

liquid

heat

Package Design (con’t)Package Design (con’t)

Rule of thumb• Solid heat sink JA 1C/W

– Thermal Resistance• DRAM leakage: J 80C

• 1280W, 40C JA 0.03C/W

A Physical View of Computer Architecture 22

Thermal LimitsThermal Limits

DRAMs

processors

router

42.5W CPU+ 2W L2 cache

(12W total)

3W logic +1W substrate

(4W total)

41W activesimultaneously

(4W total)

Major Design Implications– PC Processor

10-30W– First Order

Constraint• MHz• Latencies

Observation• Activity vs. State Retention Density

A Physical View of Computer Architecture 23

Package Design (Package Design (con’tcon’t))

Rule of thumb• Standard 15A, 110V AC outlet 1300W

first stage

second stage

120A 12V DC

1,280A 1V DC-10%

-5%

15A 110V AC

10% variation Power Supply

– PC 10¢ / W– Server 30¢ / W– Exotic $1 / W

A Physical View of Computer Architecture 24

flex signal PCB

Package Design (con’t)Package Design (con’t)

Cable Connectors– Serial, e.g. USB– 64 per side

Finger Access, Airflow

Rule of thumb• Connector 0.3 in2 panel, 0.7 in2 clearance

1 inch

A Physical View of Computer Architecture 25

Memory LatenciesMemory Latencies

R R

FSB addrNB FSBdata NBglobal cell array DQ

global cell array DQ

RR S R S Rglobal cell array DQ

RR S R S Rglobal cell array DQ

Rule of thumb• PCB 10cm/ns (5ns/foot), coax 20cm/ns

BT 3m cable RSRB T

PC133“3-2-3”

82.5ns

77.5ns

22ns29

58ns

3m cable B TR RS BT

76ns

176ns

Stack

Substrate

Cable

PCB trace transceiver

substrate

37.5ns

A Physical View of Computer Architecture 26

Memory Latencies (con’t)Memory Latencies (con’t)

Benefits– Performance

• Cache miss penalty

– Robustness• Global vs. local memory 30%• Remote access 3 local

– Marketability• Minimize application speed variance• “No surprises”

A Physical View of Computer Architecture 27

High LightsHigh Lights

Scalability– Partial node ... 64-node supercomputer

Performance– Latency

• PC133-timing to 1 TBytes

– Bandwidth• 32B / stack / cycle on substrate

• 1/6B / stack / cycle via cable

A Physical View of Computer Architecture 28

High Lights (con’t)High Lights (con’t)

Risk Management– Not liquid immersion; no pumps, hoses– Configurable for 110V outlet– Substrate uses ordinary silicon process

Taken Risks– Stacking chips not mainstream– Wirebond spring recent invention– Heat pipe reliability

A Physical View of Computer Architecture 29

SummarySummary

Architecture of a Large Computer– Performance– Materials– Mechanical Assembly– Thermal Management– Power Supply

Many More Issues...– Architect responsible for everything, even if s/he

doesn’t know anything about it!