inphi moves big data faster · 1/13/2021 · • electrical input: 8 x 26.5625 gbaud pam 4, ieee...

R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 1Inphi Copyright

Inphi Moves Big Data Faster

Next-Gen Data Center Interconnects: The Race to 800G

Radhakrishnan Nagarajan, Ilya Lyubomirsky

Jan 13, 2021COBO Webcast


Data center interconnect evolution ➔ hierarchical to flatter architectures

• Elimination of layers

• Very high-speed interconnects between layers

• Elimination/minimization of over-subscription

• Datacenters distributed over large geographic distances

- Source: Microsoft

Interconnect-rich

Machine to machine traffic dominates

High radix connections


Why do we care about the datacenter radix?

Switch Generation Radix = 32 Radix = 64 Radix = 128

12.8T 400G 200G 100G

25.6T 800G 400G 200G

51.2T 1.6T 800G 400G

Network flattening

Optical

interconnects

-Source: Facebook


Ubiquitous 2x speed scaling

60Gbaud, QAM 16, 1λ

53Gbaud, PAM4, 4 λ’s

2x λ’s2x baud rate

Higher order

modulation format

… analog bandwidth … power, size, cost, not scalable

… complexity, SNR

400G, 4λ’s ➔ 800G, 8λ’s

QAM 256PAM 16

Engineering trade-offs

QAM 64, 90Gbaud

400GPAM 6, 90Gbaud


Technology choices in data center interconnects

IEEE 802.3

PSM4, CWDM4

NRZ

25G/λ,CWDM

PAM4

Coherent

100G to 600GDWDM

Coherent

QPSK/8QAM/16QAM

100G/200G/400GDWDM

Coherent

QPSK/8QAM

100G/200G/300GDWDM

Inside DC DCI-Campus/Edge DCI-Metro DCI-Long Haul

View at 100G

IEEE 802.3

LR4

NRZ

25G/λCWDM

IEEE 802.3

DR4, FR4

PAM4

100G/λCWDM

Coherent, OIF/IEEE/ITU-T

QPSK/8QAM/16QAM (400G ZR)

100G/200G/400G Pluggable ModuleDWDM

View at 400G

Distance (km)0.1 2.0 10 100 600 10000

IEEE 802.3

DR4, FR4

PAM4

200G/λCWDM

Coherent, OIF

QPSK/8QAM/16QAM (800G ZR)

100G/200G/400G Pluggable ModuleDWDM

View at 800G

CWDM LR

coherent

Distance

optimized,

low power


Inside data center PAM 4: 400G evolution to 800G

400G ➔ 4 λ 800G ➔ 8 λ 800G ➔ 4 λ

Host side interface 50G PAM4 100G PAM4 100G PAM4

Number of host side high speed lanes 8 8 8

Line side interface 100G PAM4 100G PAM4 200G PAM4

DSP CMOS node 7nm 7nm / 5nm 5nm

PAM4 DSP die area 1.0 2.0 (7nm) 1.X

PAM4 DSP power 1.0 2.0 (7nm) 1.4

Other optics, TIA, Driver power 1.0 2.0 1.X

Optics + electronics BW requirements ~ 40GHz ~40GHz ~65GHz

Optical laser source power 1.0 2.0 1.X

Smallest pluggable form factor QSFP-DD/OSFP QSFP-DD/OSFP QSFP-DD/OSFP

200G/λ is necessary to 1.6T modules


Inside data center: Coherent vs PAM


Pluggable module form factors for data center interconnects

QSFP-28 QSFP-DD 2

OSFP

CFP2

• QSFP-DD maintains the same width as the QSFP-28 and is backward compatible.

• OSFP is a slightly wider form factor.

• A 1RU (1.75” height) switch chassis, in a 19” (width) rack, will accommodate 32 modules of QSFP, QSFP-DD and OSFP form factors.

100G PAM 4 DCI module

400G QAM 16 DCI module

CFP2 module for size comparison


High speed optical interconnects enabled by silicon photonics

High speed Mach Zehnder modulator High speed Ge photodetector

Polarization beam splitter • Performance of key components in the integrated Silicon

photonics chip across C band.

• High speed Mach Zehnder modulator

• High speed Ge photodetector

• Low loss, high extinction ration polarization beam splitter.


Between data centers: silicon photonics based 400G ZR coherent modules

• QSFP-DD module ➔ similar form factor to DR4 and FR4 modules

• Advanced 7nm CMOS DSP, Canopus

• Advanced silicon photonics node from Bi-CMOS foundry

• Electrical Input: 8 x 26.5625 Gbaud PAM 4, IEEE 802.3bs

• Optical Output: 1 x 59.8 Gbaud QAM 16, 1 λ

• 80km-120km single span reach


Between data centers: silicon photonics based 400G ZR performance at 80km

• Polarization scrambled at 50krad/s

• < 0.5dB polarization penalty

400G switch

Multi-channel

spectrum


200G/λ: Modulation,

Equalization and FEC


Optical standards for data center interconnect

Standard ModulationBaud Rate

(Gbaud)

Wavelength

(nm)

Distance

(m)FEC

FEC NCG

(dB)

FEC Latency

(ns)

400GBASE-SR8 PAM4 26.56 850 100 RS(544,514) 6.8 100

400GBASE-FR8 PAM4 26.56 1300-LWDM 2,000 RS(544,514) 6.8 100

400GBASE-LR8 PAM4 26.56 1300-LWDM 10,000 RS(544,514) 6.8 100

400GBASE-ER8* PAM4 26.56 1300-LWDM 40,000 RS(544,514) 6.8 100

400GBASE-SR4* PAM4 53.125 850 50 RS(544,514) 6.8 100

400GBASE-DR4 PAM4 53.125 1300 500 RS(544,514) 6.8 100

400GBASE-FR4 PAM4 53.125 1300-CWDM 2,000 RS(544,514) 6.8 100

400GBASE-LR4-6 PAM4 53.125 1300-CWDM 6,000 RS(544,514) 6.8 100

400GBASE-LR4-10* PAM4 53.125 1300-CWDM 10,000 RS(544,514) 6.8 100

800GBASE** PAM4 (6) ~112-114 1300-CWDM ~1-2kmHigher Gain

FEC~9-10 < 300

* Under development

** IEEE 802.3 Beyond 400GbE Study Group starting work in Jan. 2021


Source: 800G Pluggable MSA, White Paper, “Enabling the Next Generation of Cloud & AI using 800 Gb/s Optical Modules”

▪ Transmitter analog bandwidth limitations and electrical reflections (see example below)

▪ Laser RIN noise (level dependent noise) - impacts PAM6 more than PAM4

▪ Fiber chromatic dispersion – penalty scales quadratically with baud rate

▪ Polarization mode dispersion, first order effect due to differential group delay (DGD)

▪ Receiver TIA noise, nonlinearity, and analog bandwidth limitations

Driver Laser/Modulator PD/TIA ADCDAC

S21 (dB)

Key challenges for 200G/λ PAM optical systems


PAM Rx DSP architectures for beyond 400GbE

ADC 10-30 tap FFE1 tap

DFE

FEC

Baud Rate

1/T

Analog Digital

PLL DCO TEDLF

Err Gen

LMS

MLSD

Mux

LMS

AFE

Slicer

1. High Speed ADC

enables

DSP Architectures

2. FFE, DFE, and

MLSD for stronger

EQ

3. Leverage DSP soft

information for

higher coding gain

FEC


Concatenated Codes: ADC Enables Soft Decision Inner Code

Source: Frank Kschischang, “Introduction to Forward Error Correction,” OFC Short Course, 2018

KP4

(e.g., Host FEC)

KP4

(e.g., Host FEC)Inside Module Inside Module


Low latency higher gain FEC simulation results


Baud Rate:

113Gbaud, PAM 4

90Gbaud, PAM 6

Optical System:

RX OMA = -4 dBm

Laser RIN = -145 dB/Hz

ER = 5 dB

TIA NEP = 12 pA/sqrt(Hz)

FEC Limit

Comparison of modulation formats for 200G/λ


>FEC threshold >FEC threshold

PAM4

113.3GBaud

PAM6

90.7GBaud

PAM4 vs PAM6 model with realistic component models


Tx chirp

FR4 2km

200G/λ PAM4 chromatic dispersion penalty


Conclusions

▪ Next generation 800G data center optical interconnects should be designed considering the

unique application requirements of data center operators.

▪ The choice of 200G/λ modulation format, DSP architecture, and FEC scheme requires careful

analysis on the engineering tradeoffs for analog component technology and optical channel

specifications.

▪ Optical system simulations show PAM4 200G/λ is a feasible technology for 800G-FR4/DR4,

synergistic with low latency AI, break out, and copper (DAC) applications.

▪ 800G coherent technology may be most useful in the longer reach applications such as 800G

for 40km, where coherent dispersion tolerance and lower fiber loss at 1550 nm are the key

advantages.


Inphi Moves Big Data Faster

inphi moves big data faster · 1/13/2021 · • electrical input: 8 x 26.5625 gbaud pam 4, ieee...

Documents