rev2 internet2 – jt february 2007 1 implementing 100 gigabit ethernet: a practical guide joel...

1

rev2 Internet2 – JT February 2007

Implementing 100 Gigabit Ethernet: A Practical Guide

Joel GoergenVP of Technology / Chief Scientist

2

Internet2 - JT February 2007

Special Note Regarding Forward Looking Statements

This presentation contains forward-looking statements that involve substantial risks and uncertainties, including but not limited to, statements relating to goals, plans, objectives and future events. All statements, other than statements of historical facts, included in this presentation regarding our strategy, future operations, future financial position, future revenues, projected costs, prospects and plans and objectives of management are forward-looking statements. The words “anticipates,” “believes,” “estimates,” “expects,” “intends,” “may,” “plans,” “projects,” “will,” “would” and similar expressions are intended to identify forward-looking statements, although not all forward-looking statements contain these identifying words. Examples of such statements include statements relating to products and product features on our roadmap, the timing and commercial availability of such products and features, the performance of such products and product features, statements concerning expectations for our products and product features, and projections of revenue or other financial terms. These statements are based on the current estimates and assumptions of management of Force10 as of the date hereof and are subject to risks, uncertainties, changes in circumstances, assumptions and other factors that may cause the actual results to be materially different from those reflected in our forward looking statements. We may not actually achieve the plans, intentions or expectations disclosed in our forward-looking statements and you should not place undue reliance on our forward-looking statements. In addition, our forward-looking statements do not reflect the potential impact of any future acquisitions, mergers, dispositions, joint ventures or investments we may make. We do not assume any obligation to update any forward-looking statements. Any information contained in our product roadmap is intended to outline our general product direction and it should not be relied on in making purchasing decisions. The information on the roadmap is (i) for information purposes only, (ii) may not be incorporated into any contract and (iii) does not constitute a commitment, promise or legal obligation to deliver any material, code, or functionality. The development, release and timing of any features or functionality described for our products remains at our sole discretion.

3


Per IEEE-SA Standards Board Operations Manual, January 2005

At lectures, symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.

4


Acronym Cheat Sheet

CFI – Call for Interest DWDM – Dense Wavelength Division Multiplexing EMI – Electro-magnetic Interference Gbps – Gigabit per Second HSSG – Higher Speed Study Group ITU – International Telecommunications Union IETF – Internet Engineering Task Force JEDEC - Joint Electron Device Engineering Council MAC – Media Access Control MDI – Medium Dependent Interface MSA – Multi Source Agreement OIF – Optical Internetworking Forum PCS – Physical Coding Sublayer PMA – Physical Medium Attachment PMD – Physical Medium Dependent PHY – Physical Layer Device SERDES – Serialize / De-serialize SMF / MMF – Single Mode Fiber / Multi Mode Fiber Tbps – Terabit per Second WIS – WAN Interface Sublayer XGMII – 10 Gigabit Media Indpendent Interface

5


IEEE802.3 HSSG

My thoughts on where we are …..

6


Current IEEE802.3 HSSG Objectives as of January 2007

To date, the objectives within the study group are:– Support full duplex only– Preserve the 802.3 / Ethernet frame format at the MAC

Client Service Interface– Preserve the min and max FrameSize of current 802.3

standard– Support a speed of 100Gbps at the MAC/PLS Interface– Support at least 10km on SMF– Support at least 100m on OM3 MMF– Support a BER better then or equal to 10^-12 at the

MAC / PLS Service Interface– Support at least 40km on SMF

7


Upcoming IEEE802.3 HSSG March Plenary

Looking for data to support the Broad Market Potential of 100Gbps.

Looking for data to support the market for 40km on SMF.

8


Shipping an IEEE802.3 HSSG Vendor Compliant Product

Key architectural points need to be addressed from the MAC to the PMD.

Likely to be mid 2009 or later before the standard is complete enough to ensure compatibility between vendors.

MAC = Media Access ControlMDI = Medium Dependent InterfacePCS = Physical Coding SublayerPMA = Physical Medium AttachmentPMD = Physical Medium DependentWIS = WAN Interface Sublayer XGMII = 10 Gigabit Media Independent Interface

LLC (Logical Link Control) or other MAC Client

MAC Control (Optional)

MAC

Higher Layers

64B/66B PCSXGXS

Reconciliation Sublayer (RS)

XGXS

XGMII XGMII

XAUI

XGMII

64B/66B PCS

XFI

XSBI

WIS

PMA

PMA

PMD PMD

MDI MDI

MEDIUM MEDIUM

10GBASE-W 10GBASE-R

Optional XGMII

Extender

Optional Interface

Not specified byIEEE 802.3

Optional Interface

9


Developing a Standard

Call for Interest

Study Group

Task Force

Working Group Ballot

Sponsor Ballot

Standards Board Approval

Publication

Feasibility and Research

Ideas From Industry Industry Pioneering

1 Year

HSSG is here

Ad Hoc Efforts

CFI July 18, 2006

Q4 ‘07

’09 – ‘10

Force10 delivers 100G Force10 delivers 100G Ethernet SystemEthernet System

10


100Gbps Architecture

Line Card

Switch Fabric

Back Plane

Power System

11


Thoughts for a 100Gbps Line Card

NPU

CAM 200 MSPS

140

Lookup DataBase

SRAM 400 MHZ DDRII+

50 72

Lookup DataBase SRAM 400

MHZ DDRII+

25 36

140

CAM 200 MSPS

10 x

CEI-11G-SR

Inter laken/SPI-S

Ingress Packet Parsing

Ingress Lookup

Ingress Packet

Edit10 x

CEI-11G-SR

Inter laken/SPI-S

Inter laken/SPI-S Egress

LookupIngress Packet

Edit

100g MAC and Phy

Fibre Inter laken/SPI-S

Ingress L ink List SRAM

400 MHZ QDRII+

50 72

Ingress Buffer

SDRAM 1 Ghz DDR

123 256 32

10 x

CEI-11G-SR

Inter laken/SPI-S

Ingress Buffer

SDRAM 1 Ghz DDR

123 256 32

Egress

L ink List SRAM

400 MHZ QDRII+

50 72

16 x

CEI-11G-LRTM

Clock, reset, PCI Express, Test

Pins

100

Clock, reset, PCI Express, Test

Pins

100

12


Thoughts for a 100Gbps Line Card

600 Mhz x 192 bit datapath is barely feasible in 90 nm. Need to study 65nm feasibility and pick the right width and clock speed.

Memory components listed in the line card diagram are available now or definitely before 2008.

32 x 10Gbps CEI serdes are possible in few 90 nm ASIC technologies.

Feasibility of Mac is well documented by belhadj_01_1106.pdf.

NPU Signal Pin Count 643 in this example including SERDES pins

Traffic Manager Signal Pin Count 1269 in this example including SERDES pins. May have to be implemented using two chips to reduce the pin count.

Die Size of the above chips are application dependent. For a hardwired NPU,TM 65 nm may be the right process.

13


Thoughts for a 100Gbps Front End

Optics – Short Reach (100m)– 10 by 10Gbps

Optics – Long Reach (10km)– 4 by 25Gbps– 5 by 20Gbps

Optics – Extended Reach (40km)– 4 by 25Gbps– 5 by 20Gbps

14



CEI-11G

CEI-11G

“A” Possible Assembly

PMDMUX /

SERDES

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

MAC

ReceiveOPTIC

TransmitOPTIC

++20G

++20G

++20G

++20G

++20G

++20G

++20G

++20G

++20G

++20G

15



0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-20-19-18-17-16-15-14-13-12-11-10

-9-8-7-6-5-4-3-2-10

Frequency (Hz)

SD

D21

Mag

nitu

de (

dB)

XFI

Note - XFI is only specified to 7 GHz.Curve shown has been extrapolatedto 15 GHz using XFI equation

SDD21 (dB) = (-0.1- 0.78 *f ^(1/2) – 0.74* f)

• XFI based on 8 inches + 1 connector for fr-4 circuit boards

• Applications suggest 12 inches + 1 connector for fr-4 circuit boards

• See proposed model

16


Thoughts for a 100Gbps Switch Fabric

TransmitMUX /

SERDES

++20G

Back Plane

From TrafficManager N

ReceiveMUX /

SERDES

Data Alignment / coding

DigitalCrossbar

Data Coding

CEI-11G

TransmitMUX /

SERDES

ReceiveMUX /

SERDES

++20G

Back Plane

To TrafficManager N

++20G

++20G

++20G

++20G

++20G

++20G

++20G

++20G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

Switch Fabric

Switch Fabric ASIC

TransmitMUX /

SERDES

++20G

ReceiveMUX /

SERDES

Data Alignment / coding

++20G

++20G

++20G

++20G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

Data Coding

CEI-11G

TransmitMUX /

SERDES

ReceiveMUX /

SERDES

++20G

++20G

++20G

++20G

++20G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

CEI-11G

To TrafficManager 1

From TrafficManager 1

17


Thoughts for a 100GbpsSwitch Fabric

600 Mhz x 192 bit data path is barely feasible in 90 nm. Need to study 65nm feasibility and pick the right width and clock speed for a digital cross bar, multiple ASIC implementation.

32 x 10Gbps CEI serdes are possible in few 90 nm ASIC technologies. Need to study scaling 500 x 10Gbps SERDES across multiple ASICs.

Die Size of the above chips are application dependent. For a hardwired switch fabric, 65 nm may be the right process.

Conceivable to see Master / Slave ASIC Architecture.

18


Thoughts for a 100GbpsBack Plane

BackplaneSERDES

Traces

Line Cards--GbE / 10 GbE

RPMsSFMsPower Supplies

Data Packet

Mid Plane applications have complexities with holes and connectors that make it incompatible for HSSG systems.

19


Thoughts for a 100GbpsBack Plane – Long Reach 30in.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-70

-60

-50

-40

-30

-20

-10

0

Frequency (Hz)

SD

D21 M

agnitude (

dB

)

Proposed CEI25 ChannelIEEE 802.3ap Informative Channel

SDD21 = -20*log10(e)*(b1*sqrt(f) + b2*f + b3*f^2 - b4*f^3) b1 = 1.25e-5 b2 = 1.20e-10 b3 = 2.50e-20 b4 = 0.95e-30f = 50Mhz to 15000Mhz

• Commercially available resin-based laminates exist to meet these requirements (see OIF2006.097.00)

• Proposed CEI25 Channel Model (see OIF2006.047.00) – Chip vendors would like to see 9dB better channel loss.

20


Power System:Thoughts for a 100Gbps Line Card

Architecture:– Clean trace routing.– Good power noise

control means better than...

– Analog target 60mVpp ripple

– Digital target 150mVpp ripple

– Excellent SERDES to connector signal flow to minimize ground noise.

– Best choice for 100Gbps systems.

Media

ForwardingEngine

Ba

ck

pla

ne

Opticalor Copper

Media

Reserved for Power

NetworkProcessor

SERDES

SERDES

21


Power System:Thoughts for a 100Gbps Fabric

Architecture:– Clean trace routing.– Good power noise

control means better than...

– Analog target 30mVpp ripple

– Digital target 100mVpp ripple

– Excellent SERDES to connector signal flow to minimize ground noise.

DigitalCross Bar

SERDES

SERDES

Reserved for Power

22


Power System:300VDC to 500VDC System

DC PSU-A

Vin ~VDC

+

-

F1

CB1

CB2

PEM-A

PEM-B

Vin ~VDC

-

F1

Backplane

-VA

-VB

RTNA

RTNB

Inrush and soft start circuit

+

V+

V-

on/off

DC PSU-B

V+

V-

System Board

on/off

V+

V-

on/off

Vo2

Vo1

Vo3

F2

F3

F3*

F4*

F5*

RTN

RTN

RTN

23


Power System:300VDC to 500VDC System

Current DC systems require large gauge wire. Reducing the current capacity will allow smaller gauge wires.

Smaller gauge wires allow ease of use, yet still can carry the 10kw to 15kw required for a 100Gbps system.

Future AC systems may have to go to 480VAC to allow for smaller gauge wires.

24


Industry System Port Count Cycle

2002 2004 2006 2010*2008

100 GE Standard In Development

GE 100’s Ports > 1000 Ports

10 GE 10’s Ports > 100’s Ports100’s Ports

10’s Ports

25


Conclusions

Establish optical interfaces

Establish electrical interfaces, including circuit board trace characteristics for both the front end optics / electrical, and the back plane electrical

Study the ASIC requirements for the Network Processor Unit, the Traffic Manager, and the Switch Fabric blocks.

Study the power subsystem.– Cooling and EMI have to be a part of this.

26


Thank You

27


IEEE 802.3 HSSG Reflector and Web

To subscribe to the HSSG reflector, send an email to:

[email protected]

with the following in the body of the message:

subscribe stds-802-3-hssg <your first name> <your last name>

end

SSG web page URL:

http://grouper.ieee.org/groups/802/3/hssg/index.html

John D’Ambrosia, Chair IEEE802.3 HSSG– [email protected]

rev2 internet2 – jt february 2007 1 implementing 100 gigabit ethernet: a practical guide joel...

Documents

forwardlooking statements

forward looking statements

statements of historical

product features

product roadmap

internet2 jt

future operations

future revenues