1Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
PROJET DAQGEN
DAQGEN
NEBULA
IDROGEN
White Rabbit
2Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
ARCHITECTURE
Standard : xTCA for Physics Basée sur du matériel du commerce Contrôleur de châssis : MCH de N.A.T Lecture des données par le Backplane :PCIe 4x Gen3 ou Eth10G.
Transfert des données : Eth10G, PCIe-over cable (industriel) 100GEth (developpement DAQGEN)
Configuration : 1GEth par IP bus.
3
Carte NEBULA
4Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Carte NEBULA
• MTCA 4.0 standard, Double-width, full size AMC.
• FPGA : 5AGTMC7G3F31• Stand alone mode (power)• ADC 2 channels 1GSPS.• White Rabbit compliant.• On board configuration (µC)• Very low noise synthesizer PLL synthesizer cleaner (LM04828)
• Front panel : WR SFP+2x SFP+ 10GbEth
• Backplane connectivity :Gbe IP bus,PCI 4x Gen3, IPMB, CLK & trigger lane.
10Gb
LMK04828
µC ATMEGA
Power CTRLDS1014
IPMI
DataSynch & config
SPI I2C
SPI
FPGA ctrl/cde & conf
SPI
SSRAM512K
Flash x 21Mb
EPLDMAXV
ADC 0810202x 1GSPS Input bandwidth :
2Ghz
Data/config
1Gb
config10Gb
WhiteRabbit
IPMB
WR
IPBus
PCIE x4
Eth
Eth
5
WR componentsPLL Cleaner Carte NEBULA
6
f =490MHz 4℮6 samples
f =990MHz 4℮6 samples
f =1990MHz 4℮6 samples
f =1490MHz 4℮6 samples
Carte NEBULA, Test ADC
7
Carte NEBULA, systéme de test WR
8
Carte NEBULA, Test PPS
• En collaboration avec le laboratoire SYRTE (Obs Paris INSU).
• 400 Fs apres 1000s et 100 Km fibre• Tests à venir :
• long période d’analyse (Etude des dérives lentes
• Reproductibilité de l’ IP WR enfonction des firmware.
• Modification des fréquences de la DDMTD.
• Modification de la soft PLL
9
Carte IDROGEN
10Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Carte IDROGEN
• MTCA 4.0 standard, Double-width, full size AMC.
• FPGA : 10GX027H4F34• Stand alone mode (power 12v)• HighPinCount FMC slot.• White Rabbit compliant.
• Front panel connectivity : WR SFP+QSFP+ 40G, USB
• Backplane connectivity :1Gbe IPbus,PCI 4x Gen3, IPMB, CLK & trigger lane.
● RTM connector : J30.● Low cost
FM
C+
Eth-40G
Eth-1G
PCIe 4x Gen3/ETH 10G
AM
C c
onn
ecto
r
white-Rabbit
Eth-1G
QSFP+
SFP+
Data transfert
4
80 diff
Config. & CTRL
Eth-1G
Serial link 1G
IPBus
Data transfert &Config. & CTRL
8 GX
4
1
1
1
RT
M
28 diff
Clk synthesiserJitter cleaner
ARRIA 10
11Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Carte IDROGEN
ARRIA 10 SX10AX027H4F34
IP bus Ethernet 1G
PCIEx 4x Gen3
IPMB
ATMEGA128EPCQ
SFP+
Ethernet 40G
RTM
WREthernet 1G
White rabbitEthernet 1G
Serial link 40G
spi
Uar
t
Uar
t
PPS / trig.Ext
MAX10
I2C
CypressFX2LP
JTA
G
AS
pro
g
LMK04828LT10104
CLK.Ext
CLK
54
1606
8
I2C SPI
QSFP+
I2C
I2C
USB
IDROGEN board
JTAG
SI5338
I2C
CLK
4 FM
C+• On board configuration (µC)
• Very low noise synthesizer PLL synthesizer cleaner (LM04828) for WR clkand derived clkl.
• Dedicated PLL for serial links● Integrated USBBlaster II.● FPGA configuration : Active serial,
IP bus.● External connectivity : PPS, Trigger,
Ext CLK.
12
VCCP, VCC
MAX8517 PLL 0.7a maxStand-alone mode
LTM4627
LTM4627
1,03v
AMC con. 12v
EN6347 Filter
VCCR_GXB
VCCT_GXB
1,8v
VCCPGMVCCBATVCCIO
VCCH_GXB
3.3v
0,9v Seq 2
Seq 3
WR
Filter
3.3v
Iout 15A
Iout 15A
3.3V Power-management
Seq 1
MAX8527
3.3v
Max 17545Ena
VCCA_FPLL
Filter
Other components
EN6360
Seq 4
Filter
Filter
12V FMCPower con. 12v
Max 17545 3.6v
3.3V FMC
1.8V FMC
12V RTM
Seq 5
Filter
Carte IDROGEN, arbre des alimentations
13
10AX027F35
DACAD5662
DAC 16bAD5662
VCTCXO25MHz
VCXO25MHz
DACAD5662DAC 16bAD5662
DAC SPI
LMK_CLK0
WR DMTDCLK T ou B
RFCLKL 10Gb
LMK04828
FMC_CLK1
GBTCLK
100MHz
FMC_CLK0
100 MHz
MAX 10
CLKB
REFCLKR
CLKT
LMK_CLK1
AMC_clk2
REFCLK&25 MHz
25 MHz
CLK_SMA
CLKT
LMK_CLK2clk_free
ref_in1
ref_in0
XT
CA
co
nne
ctor
AMC_ clk2
FMC clk2
FMC clk2
AMC_clk1
PCIECLK
SI 5338
156.25MHz
50 MHz
125 MHz
RFCLKL 1Gb
RTM
100 MHz
RTM_ clk2
RTM clk2
Carte IDROGEN, arbre d’ horloges
14
Eth1G
FPGA Avallon bus
AM
C c
onn
ecto
r
Eth1G
JTAG
IP_bus
MAX 10Cypress
EPCQ_L
US
B
JTA
G
FMCRTM
Swich JTAG
USB blaster II
JTA
G
AS
pro
g
RemoteUpdate IP
PCIePCIe
SFP+Serial link
MMC
Carte IDROGEN, configuration FPGA
15
FM
C
Eth-40G
Eth-1G
PCIe 4x Gen3/ETH 10G
AM
C c
onn
ecto
r
white-Rabbit
Eth-1G
QSFP+
SFP+
Data transfert
4
80 diff
Config. & CTRL
Eth-1G
Serial link 1G
IP_Bus
Data transfert &Config. & CTRL
8 GX
4
1
1
1
RT
M
28 diff
Carte IDROGEN, bus
16
Eth 1g
VCXO50Mhz
Ref clock generator
PLL 2PLL 1
LMK04828 Clock jitter cleaner
DACVCXO
25MHz
25MHzVCXO
DDMT clock generator
DAC
125MHz
DS18S20Temp &
Serial num
Endpoint
LatticeMicro32
Periph
Phy
Wishbonecrossbar
DDMT PLL
WR PTP core
Flash
FPGA
PLL
125MHz
SysCon
DDMTSoftPLL
PLL
62.5MHz
User core
125MHz
62.5MHz125MHz
https://www.ohwr.org/projects/white-rabbit
Carte IDROGEN, arbre horloge WR
17
Endpoint
Wishbonecrossbar
EB masterwrapper
Mini-NICWishbonecrossbar
RAM
Fabricredirection
LatticeMicro32
SoftPLL
SysCon
PPS
Periph
UART
Bulid ID
WishbonAvalon
Avaloncrossbar
EEPROM
Tunableoscillators
JTAGMaster
Phy
WR PTP Core
Ethernet 1GWR
Wishbonecrossbar
RAMhttps://github.com/GSI-CS-CO/bel_projectshttp://www.ohwr.org/projects/white-rabbit/wiki/WRReferenceDesign
Firmware White Rabbit
18
Phy
WR PTP Core
Data acquisition
Etht 40G
WR Eth 1GEtherbone &Avalon Bridge
WR clkPPS
TBI/serdes
Avalon MM M
WB M
SoSi
WB S
FMCAvalon MM S
SiSo
Extoscillator
Data transfer
Avalon ST So
Avalon ST Si
PCIe
Avalon MM S
Avalon MM S Avalon MM S
IP BusEtht1G
Firmware IDROGEN
19Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Carte IDROGEN
20Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
WHITE RABBIT
21Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
An extension to Ethernet which provides:– Synchronous mode (Sync-E)– Deterministic routing latency
● Sub-nanosecond synchronization in WR is achieved by using the following three technologies together:– Precision Time Protocol (IEEE1588).– Synchronous Ethernet.– DMTD phase tracking.
Open hardware (CERN )
22Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
● Synchronizes local clock with the master clock by measuring and compensating the delay introduced by the link.
● Link delay is measured by exchanging packets with precise hardware transmit/receipt timestamps.
Precision Time Protocol (IEEE1588)
23Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Synchronous Ethernet
–
● All network nodes use the same physical layer clock,generated by the System Timing Master.
● PTP is used only for compensating clock offset.● Having the same clock frequency everywhere enables
phase detector technology as the means of measuring time.
24Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Phase tracking
Measure the phase shift between transmit and receive clock onthe master side, taking the advantage of Synchronous Ethernet.
Monitor phase of bounced-back clock continuously.Phase-locked loop in the slave follows the phase changesmeasured by the master.
25Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Digital Dual Mixer Time Domaine phase detector
26Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Documentation
.
• En principe sur ATRIUM•Mais actuellement problèmes d’accès avec RENATER……….
27Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Software DCOD framwork
DCOD INTEREST
E. Legay - V. LafageALATO - NETWORK, DATA STORAGE AND PROCESSING
Server 1 Server 2
Blue producer
Blue to yellow filter
Yellow consumer
PMHPMH
Blue data
Yellow data
PMHPMH
Yellow data
CTLCTL
TL Yellow
CTLCTL
TL Yellow
External application
POSIX interface
TL Alien
External applicationor electronic device
Alien data
Access to data through PMH:• Buffer by buffer (as
Narval)• Event by event
• DCOD tools :• DCOD lancher : a process to control them all• DCOD monitor (Distributed framework)
• Implemented on further detectors : AGATA, VAMOS,….
• DCOD Framwork • PMH – Posix Memory Handler• CTL Common Transport Layer• Narval acquisition : distributed acquisition system• ENX : slow control et configuration (unified interface)
Standalone Narval capability on personnal computer
28Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Conclusion
Carte en cours de routage, test en fin d’année.
Firmware développé sur la carte d’évaluation (ATILA sce REFLEX).
Le portage de DCOD sur IDROGEN sera testé sur la carte d’évaluation par le
CSNSM.
IDROGEN Disponible début 2019.
Mezzanine FMC basée sur open hardware IDROGEN sera développée pour
projet AGATA.
29Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
Carte IDROGEN
ARRIA 10 SX10AX027H4F34
IP bus Ethernet 1G
PCIEx 4x Gen3
IPMB
ATMEGA128EPCQ
SFP+
Ethernet 40G
RTM
WREthernet 1G
White rabbitEthernet 1G
Serial link 40G
spi
Uar
t
Uar
t
PPS / trig.Ext
MAX10
I2C
CypressFX2LP
JTA
G
AS
pro
g
LMK04828LT10104
CLK.Ext
CLK
54
1606
8
I2C SPI
QSFP+
I2C
I2C
USB
IDROGEN board
JTAG
SI5338
I2C
CLK
4 FM
C+
• MTCA 4.0 standard, Double-width, full size AMC.
• FPGA : 10GX027H4F34• Stand alone mode (power 12v)• HighPinCount FMC slot.• White Rabbit compliant.• On board configuration (µC)• Very low noise synthesizer PLL synthesizer cleaner (LM04828)
• Front panel connectivity : WR SFP+QSFP+ 40G, USB
• Backplane connectivity :1Gbe IP bus,PCI 4x Gen3, IPMB, CLK & trigger lane.
● RTM connector : J30.● Integrated USBBlaster II.● FPGA configuration : Active serial,
IP bus.● External connectivity : PPS, Trigger,
CLK.● Low cost
30Daniel Charlet Journées IN2P3 Clermont-Ferrand -05-2018
1 55 Signaux 100 Ohm
2 35 GND
3 17 Signaux 100 Ohm
4 35 PWR 3.3v
5 17 Signaux 50 Ohm
6 70 GND
7 70 PWR 0,9v/12v/1,8v
8 17 Signaux 50 Ohm+PWR
9 35 GND
10 17. Signaux 100 Ohm
11 35 GND
12 55 Signaux 100 Ohm
0,460mm
75 1
100 2
100 3
90 4
75 5
64 6
75 7
90 8
100 9
100 10
75 11
954
1,4mm
Empilement