Department ofPhysics

Analysis and Modelling of a 2D OpticalInterconnect for Parallel Computing

Why Optics?•Inherent parallelism•Higher temporal and spatial bandwidths•Higher interconnection densities•Less signal cross-talk•Immunity from electromagnetic interference andground loops•Freedom from quasi-planar constrains•Lower signal and clock skew•Lower power dissipation•Larger number of fan-ins fan-outs•Potential for reconfigurable interconnects

‘Optical Highway’ carrying hundreds ofthousands of channels linking non-local

nodes with thousands of channels

AMOS ProjectG. A. Russell, J. F. Snowdon, T. Lim, A. C. Walker, J. J. Casswell, K. Symington

(Department of Physics, Heriot-Watt University)

P. M. Dew, I. Gourlay, K. Djemame

(Informatics Research Institute, University of Leeds)

Department ofPhysics

Technologies - Free Space Optics

Lens

Combination of bulk and micro lenses could be used

Code V simulations to insure aberration limited spotsize small enough to eliminate cross-talk betweenchannels. Model developed to generalise to multiplerelay stages.

Polarisation Beam Splitters (PBS)

Used to route data streams into and out of OpticalHighway.

Patterned Birefringent Plates are used toselectively flip polarisations of the beams to berouted out of the Optical Highway.

Department ofPhysics

Technologies - Smart Pixel Array

VCSEL - Light Source

Modulator - Data Transmission

Detector - Data Reception

Arrays of optical componentsfabricated in a suitable opticallyactive material are ‘flip-chip’bonded to a silicon processinglayer.

Modulators have a number ofadvantages for bonding tosilicon especially in thermaldissipation.

Silicon layer may be conventionalVLSI silicon or FPGA (FieldProgrammable Gate Array) forreconfiguration.

Silicon can be used for routing orfine grained processing

Department ofPhysics

Physical Layout for a 2D Interconnect

•Data channel hard ‘wired’

•Polarisation used to control optical path

•Wrap-around links used at ends of eachdimension to condense package andreduce optical path length

•18 node layout shown below

Department ofPhysics

Experimental Optics Set-Up

Proof-of-concept set-up to show polarisation routing used to sendsignal beams from VCSEL array to CCD cameras.

Used to help develop models of the optics and obtain perimetersused in later modelling of the system.

Input pattern

Two relay output 1.3v

Three relay output 1.5v

Department ofPhysics

Modelling

•Analytical model

•Equal Bandwidth Methodology

�Longer length links require morechannels as bandwidth decreases withoptical loss

�Requires multiple clock speeds acrossoptoelectronic interface chip

•Link bandwidth assume to be receiverlimited

•SPA silicon only used as routing / trafficcontroller

•Simulation / Experimentation required tovalidate model

��

���

�

�

�� X

X

ooptical NfB�

��

1)1(

41

910402

�

���

cpq

Loptical

Department ofPhysics

Comparison of Results

2D Optical Interconnect compared with analysis of 1D OpticalInterconnect and Aspect limited electronic interconnect assuming equal

silicon area as basis.

0.00E+00

2.00E-08

4.00E-08

6.00E-08

8.00E-08

1.00E-07

1.20E-07

0 100 200 300 400 500 600 700 800 900 1000

Processors

Late

ncy

- s 1D Optical2D Optical

1.00E+04

1.00E+05

1.00E+06

1.00E+07

1.00E+08

1.00E+09

1.00E+10

1.00E+11

1.00E+12

1.00E+13

1.00E+14

0 100 200 300 400 500 600 700 800 900 1000

Processors

Band

wid

th -

bit/s

1D Optical2D Opticalelectronic aspect limit

Department ofPhysics

Computational Issues of 2D Architecture

Architectural Issues

• High connectivity (e.g. completely connectednetwork or hyper-cube like), below•Large buffers in smart-pixel layer

Algorithmic issues

•Algorithms designed to allow communication oflarge messages (to exploit optical bandwidth)•Messages collected in smart-pixel layer prior tointer-processor communication.

Initial performance results, basedaround an implementation of theBulk Synchronous Parallel modelindicate potential for substantialperformance enhancement forcommunication intensive problems.Examples of such problems areparallel sorting, graphics and FFTs.

Combiningand re-

ordering ofmessages

Barriersynchronisation

Communication

.

.

.

P

P

P

P

Local computation

time

.

.

.

P

P

P

P

.

.

.

P

P

P

P

.

.

.

P

P

P

P

Schematic depiction of a BSP computation. Each processorperforms a local computation, and this is followed by a

communication phase (after combining and re-ordering ofdata). A barrier synchronisation is then implemented across

all the processors.

Department ofPhysics

Graphics Subsystem - An Application

Processor Nodes are now the GeometricEngine, Render Engine, Floating Point Unitand other specialised processors. Inter-processor interconnect is also ‘memory bus’.

Optics connects all nodes / engines of eachtype in one dimension and all nodes with insame unit (one of each node) in the other.

Geometric Engine Render Engine

Graphics MemoryFloating Point Unit

Geometric Engine Render Engine

Graphics MemoryFloating Point Unit

Geometric Engine Render Engine

Graphics MemoryFloating Point Unit

Geometric Engine Geometric Engine Geometric Engine

Render Engine Render Engine Render Engine

Graphics MemoryGraphics MemoryGraphics Memory

Floating Point Unit Floating Point UnitFloating Point Unit

Department ofPhysics

High Bandwidth Memory Bus - A SecondApplication

8 X DRAM128 bit @ 200MHz

with8 SPAs

1 X 128 @ 200MHz

1 SPA8 X 128 @ 200 MHz

Mux / Demux8 X128 @ 200 MHz

TO / FROM1 X 128 @ 1.6 GHz

Processor

Memory interleaving

Memory interleaving is a standard techniqueto increase memory bandwidth by using alarge number of memory banks and utilisingthe aggregate bandwidth. This becomes veryimportant as processor speeds increase or ifmultiple processors share memory.

Problem

The large number of channels requiredmakes this impractical using an electronicsolution.

Using an out-of-plane optical interconnectthe channels can be supported.

Department ofPhysics

A Massively Parallel, Optically ConnectedComputer - Bringing it all Together

Processor Processor

Processor

Processor

Graphics

Graphics Memory

Memory

Parallel computer architecture includingpossibly thousands of processors,memory arrays, graphics pipelines anddisk arrays.

The individual nodes maybe internallyoptically connected, e.g. the graphicsnodes based on the design previously.

Optical back-plain directly interfaceswith internal component optics andexternal telecommunications optics.This would create the processingresources for projects such as the ‘Grid’.

Department ofPhysics

Conclusions, Research Areas andFurther Work

•2D interconnect can provide high bandwidth, highly scalible communications for massively parallelcomputers.

•Latency penalties from optics to electronics domain changes are of set by smaller routing costs of themore direct connections available optically.

•High bandwidth required for communications bounded problems and to service the data requirements offaster processing.

Research Areas and Further Work

•Large, reliable arrays of optoelectronic components (>128x128).

•Integration with silicon technologies (bonding techniques or silicon devices). OFPGAs.

•Efficient thermal engineering.

•Simulation of real traffic patterns on interconnect.

•Topology of interconnect and components.