monday, june 2 2014tipp 2014 @ amsterdam1 a scalable gigabit data acquisition system for...

TIPP 2014 @ Amsterdam 1Monday, June 2 2014

A scalable gigabit data acquisition system for

calorimeters for linear collider

GASTALDI Franck

Grant ANR-2010-0429-01

On behalf of the electronic & software team

TIPP 2014 @ Amsterdam 2

Introduction: ILC detectors

• Method: Imaging calorimetry ~100 106 channels/detectors• Issues:

– Integration– Power consumption

• Ideas:– Detectors prototypes

• Power pulsing (1% duty cycle ~25µW/ch) allowed due to the beam structure (5 Hz spills)

– Switched on during > ~1ms of ILC bunch train and data acquisition– Bias currents shut down between bunch trains

– Data acquisition and control • A single cable for everything• Scalable architecture• Reliable protocols & simplicity

Monday, June 2 2014


Introduction: ‘generic’ DAQ

• In most cases, detectors and associated readout systems are designed, tested and approved before DAQ effort is undertaken

• Our idea for this project is to design as a ‘generic’, scalable, and a self contained system, build around commercial components where possible.

• This DAQ is then configured towards multiple

‘use-cases’. ILC calorimetry might not be the only customer

Remark : This work follows a R&D from Univ.College London, Manchester Univ and Cambridge Univ that continued at LLR-Ecole Polytechnique / IN2P3-CNRS

Monday, June 2 2014


Calorimeter DAQ: overview

Monday, June 2 2014

Machineclock

DIFs SlabsDAQ2 PC

DCC(optiona

l)

Clock& Control

Digital (Config, Control, Data)Clock & Sync

GDCC

Network card

GDCC×7

⋮×8

Optic GigE or copperDebug USB

×n layers

:

DCC(optiona

l)⋮×8

Slabs = detector unit : detector with integrated front-end electronics and sensorsDIFs: Detector InterFace, servicing the detector unitGDCC: Giga-Data-Concentrator-Card: Concentrates data, fanin/fanout for clock and control dataCCC: Clock & Control card: Fanout of clock and fast controlsDCC: Data concentrator Card: optionnal extra level of data concentration

50 Mb/s

50 Mb/s

50 Mb/s

50 Mb/s

50 Mb/s

50 Mb/s

1 Gb/s


Calorimeter DAQ: Serial Link (cont’d)

HDMI connectors between DIF-DCC-GDCC-CCC

- Commercial standard for consumer electronics

- High-bandwith connection at low cost

3 twisted pairs + 2 optional• Reference clock (50 MHz), fan-out from CCC• Data in (fast control, slow-control)• Data out (slow control, data readout)

Monday, June 2 2014


DAQ: The DIF card

• The DIF concept is generic in firmware, running on detector specific hardware– Based on low cost FPGA– Compact (73mm x 50 mm)– Control up to 10K channels

• Functionalities are simple– VFE chip management (power pulsing, SC, DAQ) with a common

interface– Local storage of SC data (Flash Ram)

Monday, June 2 2014

Architecture of the DIF FPGA


• Format : VME 6U (chassis with only J1 connector used for power distribution)• Format shared in 2 part (1/3 – 2/3)

1/3 is the mezzanine with the HDMI connectionsReliability of mezzanine by a specific Samtec connector (SEAM and SEAF series: 160 pins)Until 28 differential signals and 19 single ended

2/3 is the GDCC “heart” with the main functionalitiesBased around a Xilinx Spartan XC6SLX75 + Marvell component

• USB is used to an extra access to the GDCC (debug for example)

DAQ: The GDCC card

Monday, June 2 2014

7 x DIFs HDMI CCC HDMI

RJ45 & sfp fiber VME USB

Main part

Mezzanine part


DAQ: the GDCC card (cont’d)Functionalities:• Aggregate data from many DIF links and send it to the PC over Gigabit Ethernet link

– The PHY layers is made by a specific component MARVELL88E1111• Signaling between the DIF and the GDCC is made by 5 differential LVDS pairs in HDMI cable • Extract packet from the PC and execute the command sent (R/W register, DIF configuration

packet, fast command)• Encapsulate data from DIF in Ethernet frame and send them to the PC

Monday, June 2 2014

Gemac

lithe

Homemade

(from xilinx reference design)

Totally free

Main

Interface

(based on several FSM

And few Xilinx reference design)

DIFsLinks

(Protocol fsmser-des8b/10b)

E

T

H

E

R

N

E

T

T

O

D

I

F

CCC interface

MARVELL

component

FPGA

mclk

trig

Single architecture of GDCC card


DAQ: The GDCC card (cont’d)

Dst MAC Src MAC Ethernet Type

GDCC_type GDCC_modifier GDCC_pktID GDCC_dataLength GDCC_Data PAD CRC32

6 Bytes 6 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes Variable Min size Eth 4 Bytes

Monday, June 2 2014

GDCC frame to the PC is based on standard Ethernet format

3 kinds of frameFast-command with a special Ethernet type 0x809 (GDCC DIF)Control data with a special Ethernet type 0x810 (GDCC DIF)Read-out data with the Ethernet type 0x811 (DIF GDCC)

GDCC Header

Content of the DIF structure

Data

Example of sending data from GDCC to DIFData are sampled on rising-edge of clock

CLK

DIF SOF DIF EOF

Register packet from DIF


DAQ: GDCC improvement• Put in place an UDP interface

– Simple and fast protocol– Easy to be implemented in hardware– Does not require big resources

Monday, June 2 2014

Header ETH Data

Header IP Data

Header UDP Data

@MACdest

@MACsrc

Linktype

Portsrc

Portdest Length Checksum

Version &length Service Frame

length ID Flag Offset Time oflife Protocole Checksum IP

srcIP

dest

RX

Reveive

Header_Filter

Frame_isValid

HeaderUDP (3)

HeaderIP (2)

HeaderETH (1)

REGs

Flags

GOOD FRM

BAD FRM

Send

BufferQueuePacket switch

FIFO9x1k

FIFO9x1k

DATA_OUT

Header generator

Header RX

Header TX

Checksumheader, frame

LengthIP, UDP

MEMORISE source adressIP, MAC, Port

TX

DATAs+

Nb count words

DATA_IN(UDP frame)

DATA_IN(ETH frame)

Ethernet frame structure with UDP Header

Architecture of UDP bloc

Currently under the first tests and after 3 days of sending a command to the DIF to read some registers (~ 3.106 times), there is no error.


DAQ: The DCC card (optional)

• VME format• VME only used for the card

power supply• 1 HDMI connection for the

GDCC• Until 8 connections for the

DIFs• Identical data rate at the input

and output (50 Mb/s)

Advantage:

This card can be connected or disconnected in DAQ chain without modification of behavior.

Monday, June 2 2014

Architecture of DCC


Calicoes Software The Acquisition chain

• Ecal dedicated software suite• Based on the Pyrame

framework (LLR)– Based on XML language– Allow to prototype rapidly a on-line

system

• Multi-media distribution(files, sockets and shared memories)

• Online event-building

Monday, June 2 2014

Acquisition chain: software architecture


Calicoes Software The Control-Command

• Highly modular and distributed• Control the Ecal electronics

but also the peripheral devices (Power supply, pulse generator,…)

• Provides a high level state machine for final user

• Scripting language (Python)• Good stability

Monday, June 2 2014

Global control-command architecture


The system: beam test

This DAQ has been used on the SiW-Ecal technical prototype for two years

It has been used successfully for 4 test beams at DESY

Typical setup is : (~2.5K Channels)

10 layers of detection, 10 DIFs, 2 GDCC

Monday, June 2 2014

DAQ chassis SLAB structure

S/N > 14

250 GBytes of data have been generatedThis system has been validated for 10 Hz of spill frequency (ILC requirement is 5 Hz)

Exemple of event display1e- (5GeV)5 W plates between layers


Conclusion• The aim who was to develop a DAQ system generic in nature, using commercial

components where possible has been in most part attained• The tests had shown the ability of the DAQ to take a lot of data (~250 GB)• During the last beam test, 120000 configurations have been injected in the system in

three weeks. It remained stable during all this time.• Currently, we improve our system with the implantation of a UDP block on GDCC.

Next stepConnect the DAQ to a real calorimeter system

– 16 ASICs per ASU (under test today), will be up to 160 ASICs per layer

Perspectives for ECAL(ILC)With this actual configuration and for 100M channels ECAL for example the setup will be:

12500 DCC, 2000 GDCC and 200 PC

For reducing the number of card, the main work must be done on front end modules for easiness of integration

Monday, June 2 2014

TIPP 2014 @ Amsterdam 16Monday, June 2 2014

Thank youfor

your attention


Back up

Monday, June 2 2014


Time line

Monday, June 2 2014

Physics prototype

Technological prototype

ILD ?

Proof of concept

• Linearity• Resolution• Sensors• Very front-end

Feasibility of design options

• Compactness• Granularity• Front-end• Power pulsing• Long SLAB

Construction

• Integration• Environement• Services• Industrialization• Tooling• Project org.

S/N ~ 7.5

~24 X0, 20 cm thick~2500 m2 active detectors~100M readout channels

2004-200830 layers4000 channels

1500 channels/dm3 4000 channels/dm3 4000..10000 channels/dm3

S/N ~ 15


Slab details

Monday, June 2 2014

ASUS with 16 Asics (180 x 180 mm) 1 Si Wafer with 256 pixels of 5X5 mm2 and thickness of 325 µm

Battery charger applicationAVX BestCap BZ01After regulator

360 mm

190 mm

18

0 m

m70

mm

Slab overview


DIF card

Slow control and read-out• Sent from the DAQ/control PC as a raw Ethernet frame• Passed to/from the DIF via GDCC/DCC with the

following structure (protocol)

Monday, June 2 2014

Internally decoded frame (test pin)

DIF input:Standard packet

DIF output:Here: read out of 13x16b status registers (Reshaped into GDCC frame)

IDLE

SOF dataheader EOF

Exemple of a decoding frame at the DIF level

Exemple of a fast decodind command at the DIF level


GDCC: some plots

Monday, June 2 2014

Example of data readout

Example of Result of eyes diagram and jitter on data readout

Example of readout packet spied by wireshark

Trigger = start spill

DIF SOF

RJ ~23 ps DJ ~166ps eye width ~19.75ns

0xfcff = start spill

0xfdff = start chipdata

Data from DIF


CCC card

• Supplied by University of Cambridge in 2009

• Synchronize all sub-systems upon pre-spill warning

• Until 8 HDMI connection• Distribute asynchronous fast

trigger and/or busy signals• Capable to run stand-alone for

distribute clock (50 MHz) and spill from an external trigger

Monday, June 2 2014

monday, june 2 2014tipp 2014 @ amsterdam1 a scalable gigabit data acquisition system for...

Documents

generic daq

example daq

amsterdam5 daq

daq effort

amsterdam3calorimeter

data concentrator card

clock control card

concentrates data