I. Papakonstantinou, C. Soos, S. Papadopoulos, J. Troska, F. Vasey, S. Detraz, C. Sigaud, P.

Stejskal, S. StoreyPH-ESE CERN

1

Passive Optical Networks for Timing-Trigger and Control Applications in High Energy

Physics Experiments17th IEEE Real Time Conference, Lisboa,

May 24-28, 2010

ioannis.papakonstantinou@cern.ch

Outline

2

PON DefinitionMotivationPrototype DemonstratorProtocol of CommunicationTransceiver DesignPrototype characterizationFuture work and Conclusions

Outline

3

PON DefinitionMotivationPrototype DemonstratorProtocol of CommunicationTransceiver DesignPrototype characterizationFuture work and Conclusions

PON Definition

4

• PON is a Point-to-MultiPoint (P2MP) optical network with no active elements in the signal’s path

• Upstream and Downstream are multiplexed into ONE fiber

• In the downstream direction (Master→Slave) PON is a broadcast network

• In the upstream direction a number of slaves share the same transmission medium

• Commercial PONs run at 1.25 Gb/s or 2.5 Gb/s but 10Gb/s standards have just emerged

Cost: ~900$/OLT cost ONU ~90$/ONU. Gigabit Ethernet TRXs

Master

Slave1 Slave2

U1

U2

U1

U2

Upstream

Downstream

D1

D2

Slave N

UN

UN

...

OLT

ONUs

Outline

5

PON Definition

MotivationPrototype DemonstratorProtocol of CommunicationTransceiver DesignPrototype characterizationFuture work and Conclusions

Motivation

6

• LHC P2MP links are used to transmit timing-trigger-control information

• Current TTC system is unidirectional (optically)

• Advantages of proposed implementation

a) It is based exclusively on COTS components

b) The inherent bidirectionality of PONs means that TTC and throttle/busy networks can be merged simplifying the infrastructure and reducing the amount of connectors and cabling in the counting rooms

c) the bidirectionality of the link can be also exploited to measure and to correct for latency variations

d) FPGA based implementation offers a simple path to upgradability

e) It is a potentially clock free solution

System Specifications

7

Property PON Demonstrator

Clock rate 40 MHz (ie LHC clock rate 40.08MHz)

Max distance Up to 1000m

Splitting ratio 64

Data transferred Commands + Trigger

Latency Fixed and Deterministic

Bit rate downstream 1.6 Gb/s

Received clock jitter Able to drive a high-speed SERDES

Bit rate upstream 800 Mb/s

BW Allocation Algorithm

Round Robin

Outline

8

PON DefinitionMotivation

Prototype DemonstratorProtocol of CommunicationTransceiver DesignPrototype characterizationFuture work and Conclusions

PON Demonstrator

9

• 2 slave nodes have been implemented

• One master and one slave node are implemented on the same Virtex 5 platform.

• The second slave node is implemented on a Spartan 6 platform

• Master and slave nodes are in different transceiver tiles and clocked by uncoupled sources

OLT/TTCex

ONU1/TTCrx

ONU2/TTCrx

TTCex

TTCrx

TTCrx

Outline

10

PON DefinitionMotivationPrototype Demonstrator

Protocol of Communication

Transceiver DesignPrototype characterizationFuture work and Conclusions

Downstream Frame

Papakonstantinou et. al.

TWEPP 2009 21-25 Sep. Paris – Optoelectronics and Links Session 11

Synchronous transmission of super-frames with a period of 1625ns = 65*25ns at 1.6Gbit/s

Comma, “K”, character for frame alignment and for sync T character carries trigger info. “F” for trigger protection or

other functions D1 and D2 carry broadcast or individually addressed

information depending on the first bit of the D1 byte. “R” field contains the address of the next ONU to transmit 590.8 Mb/s are available for data downstream

Slave1

Slave2

Slave64

Upstream Frame

12

• Slave N1 receives an R character with its address and switches its laser ON

• IFG between successive emissions allows to master receiver to adapt to different bursts

• Channel arbitration is a logic built-in in the OLT

• Total BW 800Mb/s, 7.7 Mb/s are available per Slave node for pure data

Outline

13

PON DefinitionMotivationPrototype DemonstratorProtocol of Communication

Transceiver DesignPrototype characterizationFuture work and Conclusions

OLT Transmitter

14

• Latency issues at the Tx arise at clock domain crossing points• Gear Box logic ensures correct transition from 40MHz to 80MHz in the Tx-PCS• It is particularly important to bypass any elastic buffers in the data path• In GTX Tx this is achieved by advanced mode which forces PMA PLL to phase

align XCLK and RXUSRCLK clocks

Input Outpu

t

1.6Gb/s

...

2161

Frame Generator / Gear Box

...

2201

8B/10B

F I F O

P I S O

80MHz RXUSRCL

K

TI PLL

DCM

80MHz 80MHz

clock domain crossing

PMA PLL

800MHz80MHz XCLK

40MHz

Tx-PCS

Tx-PMA

Virtex- 5GTX Transmitter

Clk ref

40MHz

ONU Receiver

15

• 80 MHz parallel clk can lock on any of the 20 first edges of the 800 MHz serial clk

• That affects the order with which parallel data are exiting the SIPO

• By fitting “K” characters into the frame and identifying them in a Barrel shifter we can predict the starting point of the XCLK

Clk ref

PMA PLL

S I P O

REFCLK

800MHzRxin1.6Gb/s

2

20

1

Barrel Shifter

DCMctrl

80MHz

RXUSRCCLK

80MHz

InputCDR

80 MHz XCLK

Retimed

800 MHz Serial

...

DIV1÷10

80MHz clk

800MHz clk0 1 2 3 19. . . 1817

DCM:2

Shift value

Phase corrected 40MHz Comma

detect logic

Phase corrected

40MHz180° 40MHz

Virtex- 5 GTX Rx or Spartan-6 GTP Rx

Outline

16

PON DefinitionMotivationPrototype DemonstratorProtocol of CommunicationTransceiver Design

Prototype characterization

Future work and Conclusions

Latency Map

17

ONU TRx

Rx+

Rx-

Tx+

Tx-

OLT TRx

Rx+

Rx-

Tx+

Tx-

bias

GTX Transmitter GTX Receiver

5 ns/m

2.2 ns 2.1 ns

75ns 137.5 ns

PON TTC Introduced Latency (ns)

GTX Tx 75OLT Board 2.2ONU Board 2.1GTX Rx 137.5 (estimation)TOTAL 216.8Fiber 5 ns/m

FPGA

FPGA

ONU

OLT

Current TTC Introduced Latency (ns)

TTCex 25TTCrx 65 - 85TOTAL 90 - 110

Latency Stability Measurements

18

• Histogram of 100 reset cycles

• TI PLL power cycle → Tx reset→ Rx reset

• Stability to within <150ps

• But a better look-up-table implementation will give better results

Jitter Map

19

ONU

Rx+

Rx-

Tx+

Tx-

OLT

Rx+

Rx-

Tx+

Tx-

bias

FPGA Design

Ref clkRMS: 3.17 ps

Before DCMRMS: 17.33 ps

After TI PLLRMS: 4.48 ps

TI PLL

Ref 4

0 M

Hz

80 M

Hz

After DCMRMS: 36.72 ps O

ut

40 M

Hz

GTX Receiver

TI PLL

CDCL6010

DCM

BS

After PLLRMS: <10 ps

Latency Monitoring

20

OLT

ONU1 ONUN

U1

U2Upstream 1310nm

Feeder fiber monitoring

D1

D2

...

D2D1

ONU

D2D1

OLT

ONU1 ONUN

U1

U2Upstream

Downstream

D1

D2

...

Downstream 1490nm

Full Ranging

Rx TxTx

Rx

φ

Outline

21

PON DefinitionMotivationPrototype DemonstratorProtocol of CommunicationTransceiver DesignPrototype characterization

Conclusions

Summary

22

• A passive optical network for TTC applications has been successfully demonstrated

• Fixed and deterministic latency has been achieved in the direction of the trigger transmission within ±150ps

• Recovered clocks had <10ps RMS jitter with external PLLs

• Network is bidirectional allowing for the slave nodes to provide the master with feedback

• Also allow for latency variation detection and correction

Acknowledgements

23

This work was supported in part by the ACEOLE, a Marie Curie mobility action at CERN, funded by the European Commission under the 7th Framework Programme

24

Questions?

ioannis.papakonstantinou@cern.ch

BACK UP SLIDES

25

PONs in OSI Architecture PONs reside in the last two layers in the

OSI architecture namely Data link layer which is responsible for the

access to the medium and for error correction

Physical layer which is responsible for transmitting and receiving the information

In GPON terminology the two layers are called: G-Transmission Convergence (GTC) and Physical Media Dependent (PMD)

EPON modifies MAC layer to allow for bridging data back to the same port

System Power Budget

27

Master

Slave1 Slave2

Upstream

Downstream

Slave N...

Transceivers for PONs GPON has far more stringent

requirements than EPON For this reason GPON

components are in general more expensive than EPON

OLT needs a 1490 nm laser (usually DFB-DBR)

Burst Mode Receiver ONU needs 1310 nm F-P laser

with burst mode laser drivers PIN diode or APD at 1490 nm

AGC+

CDR

Coarse WDM

OLT

LDDFB

TOSA

Post-amp

BM ROSA (APD)

… CDR

Coarse WDM

ONU

BM-LD

F-P TOSA

Post-amp

ROSA (PIN)

Optical signalElectrical signal

1310 nm

1490 nm

Tx Enable 16 bits (12.9ns)

Tx Disable 16 bits (12.9 ns)

Total (Tg+Tp+Td) 96 bits (77.2 ns)

Guard time Tg 32 bits

Preamble Time Tp 44 bits (35.2 ns)

Delimiter Time Td 20 bits

Rx Dynamic Range 21 dB

Tx Enable 512 ns

Tx Disable 512 ns

Total (AGC+CDR)

412 ns

Rx Dynamic Range

>21 dB

GPON (1.24 Gb/s) EPON

Burst Mode Rx Decision Threshold set

29

APD

TIA Data Out

+-

LA

R

CRese

t

Peak Detection Circuit

VrefVref

Vref

P2MP Timing Parameters

24/09/2009TWEPP 2009 21-25 Sep. Paris – Optoelectronics and Links Session 30

Barrel Shifter

31

Clk refPMA PLL

S I P

O

REFCLK

800MHz

Rxin 1.6Gb/s 2

20

1

Barrel Shifter

RXRECCLK

CDR

1/10

80MHz

80MHzD I

V

80MHz clk

800MHz clk

b0 b1 b2 b3 b4 b5 …

Barrel shifter

b16

b5

b4

b3

b2

b1

b0

b19

b18

b17

. . .

“K”

b17b18

b16 . . .

Slave Rx Latency, Barrel Shifter

32

bs = 0

bs = 10

bs = 5

OLT Tx – Gear Box

33

TI PLL x2 x1

Gear Box

80MHz

...

1

32... 216

1

40MHz

TXUSRCLK

2

REF 40MHz

80MHz

40MHz

(o) (e)(o)

FT

KR

Frame Generator

IscharK

(e)

R/K/T/FD1/D2/

T/F

IscharK

FTKR

(o)

(e)

FT

ONU Rx – Comma Detect

34

DCMctrl

DCM:2

Shift value

Phase corrected 40MHz

Comma detect logic

80MHz

RXUSRCCLK

20

1

... Barrel Shifter

090

180

270

BB

B20

1

...

Rearranged data

K?

EN EN

K?

0 180

Comma Detect Logic

R/K

0

180

40 MHz

T/F

Upstream Frame• Slave N1 receives an R

character with its address and switches its laser ON

• IFG between successive emissions allows receiver to adopt between bursts

• Upstream contains 32 bytes of <5555> for CDR followed by two SFD, <D555>, bytes for frame alignment

• A 2 byte address field and data are following

• Channel arbitration is a logic built-in at the OLT

• Total BW 800Mb/s, 7.7 Mb/s are available per Slave node for pure data

35

Data Data . . . 5555 AddrK

32 B2 B

2 B90 B

OLT Burst Mode Receiver

36

Clk refPMA PLL

S I P

O

REFCLK

Rxin800Mb/s

2

20

12

20

1

Oversampling x5

Slave1

Slave2

Δφ

11 1 10 0 0 01 X 11 1 010 10 Samples0

Upstream frame (2)

24/11/200937

Slave1 Slave2

Slave1 Slave2

• Dynamic range (Pslave1/Pslave2) affects IFG and thus available upstream BW/slave

• It also affects the period between successive transmissions

IFG

Future PON

– Tri-band PONs– WDM PONs

Tri-Band PONsTri-band PON

transceivers utilize 1440nm band

They can be used in our context to separate the trigger from the control information

That can simplify protocols of communication and complexity at the OLT

39

1440 nm Tx (control)

1550 nm Tx (trigger)

1310 nm Rx

1550 nm Rx1550 nm Rx

1440 nm Rx

1310 nm Tx

1440 nm Tx

1310 nm Tx

Master

Slave1 Slave2

Wavelength Division Multiplexing PON

In a WDM PON scenario each channel (or channel group) is assigned one wavelength upstream and one downstream for communication with OLT

Benefits are higher bandwidth per channel, loss is independent from splitting ratio, less complicated scheduling algorithm at OLT, easy expansion, better delivery of services

Main disadvantage is the need for expensive WDM components such as AWGs, filters, tunable lasers / laser arrays / laser per ONU, broadband receivers etc

“Colorless” WDM PONs are developed to tackle cost issues

Receiver Array

…

WDM Rx

DEMUX

Coarse WDM

OLT

Band A

Band B

λ1, λ2, … λ16

λ17, λ18, … λ32

1 x 16 AWG …

Receiver

RN ONUs

RSOA

Receiver

REAM

λ1

λ17

λ16

λ32

λ1

λ17

λ32

λ16

Coarse WDM

Coarse WDM

λ

Upstream band

Downstream band

…

WDM Tx

SLED

3 dB λ17, λ18, … λ32

Modulated

CW