Architectural Freedomwith

Serial Interconnects

NDIA ConferenceSan Diego, CA

October 20-23, 2003

Emmanuel Eriksson – Dy 4 Systems(emmanuel.eriksson@dy4.com)

Interconnect TrendsBus vs. Switched

Device

DeviceDevice

DeviceDeviceDevice

DeviceDevice

1980’s

Device

DeviceDevice

Device

Device

DeviceDevice

Bridge 1990’s

2000’s

DeviceDevice

DeviceDevice

DeviceDeviceDevice Device

SwitchedInterconnect

Shortcomings of Parallel Buses

scale number of connections scale physically scale bandwidth support multiple traffic types support high availability

You only need to encounter one limitation to make the move to a new architecture

Buses Don’t:

Typical Embedded Distributed System Requirements

• Large amount of data movement• Large amount of data processing• Large number of processing elements• Often multiple processor types

– DSP, CPU and NPU• Wide range of I/O devices

– Sensors– Imaging and Video devices– Storage– Network Interfaces– Displays

Bus Based Architecture

CPU

IO

CARD

IO

CARD

IO

CARD

IO

CARD

IO

CARD

IO

CARD

IO

CARD

PCI Bus

CPU

Bus Bridge

Memory

SCSIEthernet

USB

Sensor

Interface

Bus

Bridge

• Single address map• Limited to a single processor or single bus owner at a time• Difficulty scaling processing requirements with I/O needs

LAN Based Architecture

• Internet Protocol is a sophisticated protocol that is aimed at a complex problem:

Interconnect hundreds of thousands of computers

– Handle extreme latencies– Handle lost packets– Handle corrupted packets– Handle out-of-order packet arrival

Limitations of Current Architectures

• Difficulty balancing computational power with I/O requirements– No support for both processor-to-processor and processor-to-device

communication (Ethernet and bus architectures can do either but not both)

• “Lossy” LAN transport do not meet lossless data collection requirements without additional software– Ethernet will drop packets if congestion occurs

• High latency or limited bandwidth scaling– Moving a 64 bytes packet takes > 100 microseconds with Ethernet versus a

couple of microseconds with a switch fabric

• Limited Class of Service capability– Ethernet has no separate queuing or buffering for CoS– Buses don’t have any CoS

Serial Switched InterconnectsScalability, Density, Flexibility Supports “disaggregated” and “virtual” systems with highly flexible upgradeability

High Availability & ReliabilityFailures detected, reported & managed.Lossless fabric with CRC protection.One failure can’t take down system

Low Cost Leverages PCI infrastructure and a wealth of products with lean, protocol-agnostic switching.

High Performance Unified Fabric Multiple types and classes of traffic supported on low latency, high bandwidth interconnect.

Device

DeviceCPU/Device

CPU/Device

DeviceCPU/Device

DeviceCPU/Device

SwitchedInterconnect

StarFabric & PCI Express Advanced Switching (AS)

• Scalable, Extensible Serial Switched Interconnects– Lean, High Performance, Low Cost Switching Architectures– Layered Approach Promotes Efficient Modularity

• StarFabric– Open standard switched interconnect (StarFabric Trade Association)– Based on 622Mb/s SERDES technology

• 4X data bandwidth of 2 Gb/s• Links can extend beyond single chassis up to 10 meters through standard

CAT5 cabling– In production since Spring 2002

• PCI Express - Advanced Switching (AS)– Based on the PCI Express™ architecture – Based on 2.5Gb/s SERDES technology

• 4X data bandwidth of 8 Gb/s– Available in late 2004

Multi-Layered Protocols

Software

Physical Point to point, serial, differential,Point to point, serial, differential,hot-plug, inter-op form factorshot-plug, inter-op form factors

Data IntegrityData IntegrityData Link

622Mb/s for StarFabric2.5Gb/s for AS

Transaction StarFabric & Advanced SwitchingStarFabric & Advanced SwitchingPacket SwitchingPacket Switching

Hardware

PCI Bus Driver PCI Bus Driver

(init, enum, config)(init, enum, config)

P-to-P DriverP-to-P Driver SF/AS DriverSF/AS Driver

Operating SystemOperating System

LegacyPeer-Peer

CRC, Link Synchronization, Auto negotiation, Link status

Data routing, Credit based flow control, Event and error handling, etc.

Example Interconnect: StarFabric

Bus

Bridge

Bridge Bridge

BridgeBridge

Bridge

Bus

Bu

s

Bu

s

Bu

s

Bu

s

With one Switch device, connect 6 Bus

segments

Switch

Redundant fabric interconnect

accomplished with a 2nd Switch device

Switch

StarFabric Frames at 2.5G Transmit and

2.5G Receive

Parallel Bus

PCI or TDM or Proprietary

New Degrees of Architectural Freedom

SF or ASFabric

Network Gateway

Processor Boards

Solid State Storage

Storage BladesStorage BladesSCSI/FCCntrl

SCSI/FCCntrl

BridgeBridge

SCSI/FCCntrl

Bridge

ProcessorProcessor

ChipsetChipset

MemoryMemory

BridgeBridge

CPU

Chipset

Memory

Bridge

GigEMAC

Bridge

DSPDSP

MemoryMemory

BridgeBridge

DSPDSP

DSP

Memory

Bridge

DSP

DSP Boards

I/O

uC

I/O Bridge

uC

I/O

uC

I/O Bridge

uC

Other IO:SensorsEO/IR VideoVoice commsEtc...

SwitchedInterconnect

I/O

CARD

CPU

SWITCH

I/O

CARD

SWITCH

STORAGE

STORAGE

CPU

1G

LAN

CPU

SWITCH

CPU

I/O

CARD

SWITCH

STORAGE

1G

LAN

STORAGE

CPU

CPU

SWITCH

CPU

SWITCH

STORAGE2

STORAGE2

10G

LAN

CPU2

CPU2

CPU2

CPU2

DSP

DSP

DSP

DSP

DSP

DSP

DSP

DSP

I/O

CARD

I/O

CARD

Adaptive InfrastructureLife Cycle Benefits

Initial Purchase Initial Upgrade

Scale capacity

Mid Life Kicker

Upgrade CPUs & Storage

Increase LAN speeds

Dy 4 System Solutions

StarFabric Interconnect

&Interprocessor

Communication Software

CHAMP-AVII(Quad PowerPC)

CHAMP-FX(Dual FPGA Board)

StarReach(PMC Carrier Board)

DMV-182(Single Board

Computer)

Advantages of Distributed SwitchingSeparate Switch Card

Requires extra slot Single point of failure Extra LRU item logistics

Switch card

X

Backplane switching

Four slots

Active circuits on backplane

Switch circuit on backplane

X

StarFabric

Passive Backplane

XX

Switching on card

No active backplane

Four slots used

Summary

• Switched Interconnects like StarFabric and PCI Express AS:– Provide low cost, high performance interconnects for multislot,

distributed architectures– Leverage the PCI/PCI-X infrastructure and volumes– Provide CPU-to-CPU and CPU-to-device communication– Reduces processor overhead and system latencies– Provides a reliable interconnect for multiple classes of service– Enables design flexibility and system disaggregation– Allows flexible fabric architecture, centralized or distributed switching

Thank You!

Contact information for follow up questions or discussions...

Emmanuel ErikssonTechnology Architect

Technology Innovation GroupDy 4 Systems Inc.

emmanuel.eriksson@dy4.com(613) 599-9199 x5462

or visit...

www.stargen.comwww.dy4.com