white rabbit - indico

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11

Erlangen12-14, October 2011 1

White Rabbit Sub-Nanosecond timing over Ethernet

H.Z. Peek ... ... on behalf of the White Rabbit collaboration

1) Alice in Wonderland, Lewis Carroll (1865)

“Oh dear! Oh dear!

I shall be too late!” 1)

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11

Erlangen12-14, October 2011

Outline

2

What is White Rabbit

Network

Building blocks

Measurement results

Applications

Conclusion

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


What is White Rabbit

Networking and timing

Based on well-known technologies / standards

Open Hardware and Open Software

International collaboration

Main features:

Transparent, high-accuracy time distribution,

Low-latency, deterministic data delivery,

Designed for high reliability,

Plug & play.

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Timing over Ethernet

History

1985

NTP (RFC 1305) sub milli-second

2002

PTP (IEEE 1588) sub micro-second

2012

White Rabbit (?) sub nano-second

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit timing distribution

Synchronization with sub-ns accuracy over fiber

A combination of:

Synchronous Ethernet (SyncE) syntonization

Digital Dual-Mixer Time Difference (DDMTD)

phase detection (j)

Precision Timing Protocol (PTP) synchronization

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Timing distribution

Averaging all clock edges • “mean phase”

Syntonization • “The adjustment of two electronic circuits or devices in

terms of frequency”

Reference

Clock

Master

SFP

Slave Rx

SFP

Tx

Tx Rx

SFP

Slave Rx

SFP

Tx

Tx Rx

j

j

High Precision

Low jitter High Precision

Low jitter

Ordinary serial data

communication

channels

Ordinary serial data

communication

channels!

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


toffset

7

j Ref

Clk Cnt

Master

SFP

Slave Rx

Tx

SFP

Tx

Rx

PTP (IEEE1588)

j Cnt

t1 t2 (= Cnt + SlaveBitSlide)

Sync message

Follow_Up Message (t1) t1 t2

t3 t1 t2 Delay_Req Message

Delay_Response Message (t4) t4 t3 t1 t2

t1 t2

Master

Clock

Time

Slave

Clock

Time

t3

t4

t4 (= Cnt + MasterBitSlide + j)

=0

Data (Packets)

Clock

Time Stamp

t2

t3 t4

t4

t1

t1

1. t2 –t1=offset + MSdelay

2. t4 –t3=-offset + SMdelay

3. MSdelay=SMdelay

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit network

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit Switch V2

Central element of White Rabbit network

Fully custom design, designed from scratch

10 SFP ports (1000Base-LX)

Capable of driving long distance Single Mode fiber

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit Node

http://www.ohwr.org/projects/spec/wiki

Simple PCIe FMC

Carrier (SPEC)

Currently available

http://www.ohwr.org/projects/spec/wiki

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit Switch V3

2 uplink ports, 16 downlink ports

Hardware just assembled.

Hardware & Software currently being tested.

Production expected Q1-2012

http://www.ohwr.org/attachments/741/White_Rabbit_Technical_Spec_05.pdf



H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Measurement test setup

http://www.youtube.com/watch?v=ZSRQEExbdq8

Hot-air gun demo:


http://www.youtube.com/watch?v=ZSRQEExbdq8

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Measurement results

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Digital to Time Converter (DTC) = Alarm Clock

Time to Digital Converter (TDC) = Time stamping

14

Applications

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit enables measurement and control

applications which are using distributed system

technologies.

Such applications may be spread over large

distances.

Data transmission delay changes are

continuously measured and compensated.

System timing of White Rabbit nodes are

synchronized with high precision.

15

Conclusions

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


White Rabbit a multi-laboratory, multi-company effort

Thank you Coming Soon!

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


How do we know the time offset

between Master and Slave?

Clock & Data coded into one stream

DC-Balance

Special code-groups / Word Alignment

1

2

3

Toffset = Total delay(Master>Slave>Master) / 2

Can we measure propagation delay using

existing serial communication channels?

Serial Communication Coding Properties:

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Measure propagation delay using

FPGA SerDes technology

Start

Lattice LFSCM25

SerDes

Lattice SC

PCI Expressx1

Evaluation Board

Xilinx Virtex-5

SerDes

Xilinx

ML507 Board

LEDs

Stop

100 Km

Fiber

19

Transmitter

SFP

Receiver

SFP

1

3

Regenerate system clock at the receiver

Using a barrel shifter for word alignment

delay known with bit clock resolution

System

Clock

x 20 System

Clock

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Master

Slave

Start

Stop

10 Km

fiber

Stop

Clock

Loopback

(DPLL)

VCXO

DAC

Time offset measurement test setup

Loopback the

recovered clock with a

Voltage Controled Xtal

Oscillator

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


0 0001 = 1 0 0000 = 0 0 0011 = 3

Resynchronization + Barrel shifting action

RxRecClk

BitSlide(4:0)

011101011000001010110111010110000010101101110101

0

Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI

Start/Stop

delay

3 1

Unit Interval

(UI)

Details : VLVnT09, October 15, 2009 in Athens

• Presentation: http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt

• Paper: http://dx.doi.org/10.1016/j.nima.2010.04.126

http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt

http://dx.doi.org/10.1016/j.nima.2010.04.126

H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


j

Reference

Clk

Fine time

Count

Coarse time

Sta

rt

Sto

p

Master

SFP

Slave Rx

Tx

SFP

RxUsrClk

TxUsrClk

TxUsrClk

RxUsrClk

1. # of system clocks

2. # bit clocks (i.e. barrel shifts)

3. Phase between Master node Tx and Rx clock

toffset

Tx

Rx

Measure time offset Bidirectional + Loopback the recovered clock

1.25 Gbps (IEEE802.3 1000BASE-X = Gigabit Ethernet)

Time offset is determined by:

For details please see Technical Report “ETR2010-01”:

http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf

1.25 Gpbs

10 .. 100 ps

20 x #bit clocks

800 ps




H.Z. Peek

Nikhef

Amsterdam

Electronics-

Technology

VLVnT11


Master time

tpdl1

tpdl1

Slave time

toffset

tdisp

dmTx dsRx

dmRx dsTx

l1

l2

Time offset and fibre dispersion

000289.2

10.9.4

14162.2

61

2

1

PP

dDV

Ptoffset

ll

ll

(tdisp = 15150 ps over 10.7 km => D(l) = l416 ps/km)

white rabbit - indico

Documents